My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?
In reinforcement learning, you must set rewards so that they are maximised when the agent achieves the goals of the problem. You should avoid trying to "help" the agent by setting interim rewards for things that might help it achieve those goals.
For card rejection, if that is part of the game (i.e. it is valid to play a "wrong" card, and you lose your turn), then either no reward, or a negative one might suffice. Probably you should go with no reward, because the punishment would be in not winning that round anyway.
If an invalid card cannot actually be played according to the rules of the game, and there is no "pass" move or equivalent, then you should not allow the agent to select it. Simply remove the action from consideration when making action selection. It is OK for the agent/environment to enforce this in a hard-coded fashion: A common way to do that, if your agent outputs a discrete set of action probabilities or preferences, is to filter that set by the environment's set of allowed actions, and renormalise.
What if you want the agent to learn about correct card selection? Once you have decided that, then it becomes a learning objective and you can use a reward scheme. The action stops being "play a card" and becomes "propose a card to play". If the proposal is valid, then the state change and reward for the round's play are processed as normal. If the proposal is not valid, then the state will not change, and the agent should receive some negative reward. Two things to note about this approach:
Turns in the game and time steps for the agent are now separate. That's not a problem, just be aware of the difference.
This will probably not encourage the agent to play better (in fact for same number of time steps, it will probably have learned less well how to win, because it is busy learning how to filter cards based on the observed features), but it will enable it to learn to propose correct cards without that being forced on it in a hard-coded fashion.
For winning a round, then you might want to reward the agent according to the game score it accumulates. Assuming that the winner of the overall game is the player with the highest score, this should be OK.
However, there is a caveat to that: If by making certain plays the agent opens up other players to score even higher, then simply counting how many points the agent gets is not enough to make it competitive. Instead, you want very simple sparse rewards: e.g. +1 for winning the game, 0 for drawing, -1 for losing. The main advantage of using RL approach in the first place is that the algorithms can and should be able to figure out how to use this sparse information and turn it into an optimal strategy. This is entirely how AlphaGo Zero works for instance - it has absolutely no help evaluating interim positions, it is rewarded only for winning or losing.
If you go with +1 win, -1 lose rewards, then you could maybe make players' current scores part of the state observation. That may help in decision making if there is an element of risk/gambling where a player behind in the scores might be willing to risk everything on the last turns just for a small chance to win overall.
