# Card game for Gym: Reward shaping

I am working on a card game for openai gym and currently I ask myself how to shape the reward function for it. One round of the game consists of each player picking a card from its hand, whereas not every card can be played depending on the card which has been played by one of the players before. For every set of played cards, there is a total order such that the player with the highest card wins the round.
In the situation in which cards are rejected I want to give the agent some reward.

In case of an invalid card, it is hard to say if that card is any nearer to one of the valid cards than any other. Also the agent should learn that this card is not playable at this point.

For completeness, the agent gets a discrete observation of everything it can remember of the game (its own cards, cards played in current round, cards played in past rounds, game mode (defines the total order of cards)). Then it should play a discrete action which either is a game mode in the beginning or a card during the round. Then it either gets a reward because its card got rejected or it gets a reward based on whether it wins the round or not. The game accounts a certain amount of points for a won round depending on the constellation of played cards in that round.

My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?

In case any more details are required, just ask for them.

My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?

In reinforcement learning, you must set rewards so that they are maximised when the agent achieves the goals of the problem. You should avoid trying to "help" the agent by setting interim rewards for things that might help it achieve those goals.

For card rejection, if that is part of the game (i.e. it is valid to play a "wrong" card, and you lose your turn), then either no reward, or a negative one might suffice. Probably you should go with no reward, because the punishment would be in not winning that round anyway.

If an invalid card cannot actually be played according to the rules of the game, and there is no "pass" move or equivalent, then you should not allow the agent to select it. Simply remove the action from consideration when making action selection. It is OK for the agent/environment to enforce this in a hard-coded fashion: A common way to do that, if your agent outputs a discrete set of action probabilities or preferences, is to filter that set by the environment's set of allowed actions, and renormalise.

What if you want the agent to learn about correct card selection? Once you have decided that, then it becomes a learning objective and you can use a reward scheme. The action stops being "play a card" and becomes "propose a card to play". If the proposal is valid, then the state change and reward for the round's play are processed as normal. If the proposal is not valid, then the state will not change, and the agent should receive some negative reward. Two things to note about this approach:

• Turns in the game and time steps for the agent are now separate. That's not a problem, just be aware of the difference.

• This will probably not encourage the agent to play better (in fact for same number of time steps, it will probably have learned less well how to win, because it is busy learning how to filter cards based on the observed features), but it will enable it to learn to propose correct cards without that being forced on it in a hard-coded fashion.

For winning a round, then you might want to reward the agent according to the game score it accumulates. Assuming that the winner of the overall game is the player with the highest score, this should be OK.

However, there is a caveat to that: If by making certain plays the agent opens up other players to score even higher, then simply counting how many points the agent gets is not enough to make it competitive. Instead, you want very simple sparse rewards: e.g. +1 for winning the game, 0 for drawing, -1 for losing. The main advantage of using RL approach in the first place is that the algorithms can and should be able to figure out how to use this sparse information and turn it into an optimal strategy. This is entirely how AlphaGo Zero works for instance - it has absolutely no help evaluating interim positions, it is rewarded only for winning or losing.

If you go with +1 win, -1 lose rewards, then you could maybe make players' current scores part of the state observation. That may help in decision making if there is an element of risk/gambling where a player behind in the scores might be willing to risk everything on the last turns just for a small chance to win overall.

• The invalid cards are the ones that are not allowed to be played according to the rules (card rejection is not part of game, I wanted to train my agent to play the rules). There is no pass move.Basically, the rules tell you which cards must be played for each situation if you have them e.g. if you have hearts and hearts is already played, you must also play one of your hearts cards. How would I make the agent only play one of the actions it is allowed to play? The set of actions would the always change and it would some need the current set in the observations in order to know what it can play – Ben Mar 9 '18 at 10:58
• @Ben: I will adjust my answer a little later, but essentially there is little reason to "teach" your agent about actions it cannot take that are enforced by the environment. The fact that your approximate evaluation model can assign probabilities to those actions should not drive you to think it is necessary to fix that - there are a whole bunch of out-of-scope actions that your agent will never consider, such as playing multiple cards at once, picking up other players' cards etc. You could in theory make your agent learn about those too, but it won't help it win anything – Neil Slater Mar 9 '18 at 11:14
• Thanks for helping me out. Just to give some more info on the invalid actions. I do not try to teach it that it can not play every single type of movement. I just do not know how I can adjust the set of valid action every time I ask the agent. I just have the same set of actions of which the agent can return one every time. Could it be that a DQN agent can not do this? – Ben Mar 9 '18 at 12:44
• @Ben: A DQN agent can do this. But as you will already have to implement the rule for valid actions in the environment, there is no need to make it a learning objective. Just have the environment return the valid list of actions to the agent, and use those to filter possible actions using usual programming logic. There is unlikely to be any benefit in performance to having the agent learn how do do this filtering itself. – Neil Slater Mar 9 '18 at 12:47
• @Ben: Just filter the actions to the allowed actions and renormalise the probabilities. You write the DQN agent, so it doesn't need to "insist" on anything. – Neil Slater Mar 9 '18 at 14:22