4
$\begingroup$

I am currently learning reinforcement learning and am have built a blackjack game.

There is an obvious reward at the end of the game (payout), however some actions do not directly lead to rewards (hitting on a count of 5), which should be encouraged, even if the end result is negative (loosing the hand).

My question is what should the reward be for those actions ?

I could hard code a positive reward (fraction of the reward for winning the hand) for hits which do not lead to busting, but it feels like I am not approaching the problem correctly.

Also, when I assign a reward for a win (after the hand is over), I update the q-value corresponding to the last action/state pair, which seems suboptimal, as this action may not have directly lead to the win.

Another option I thought is to assign the same end reward to all of the action/state pairs in the sequence, however, some actions (like hitting on count <10) should be encouraged even if it leads to a lost hand.

Note: My end goal is to use deep-RL with an LSTM, but I am starting with q-learning.

$\endgroup$
  • $\begingroup$ You have hit one of the major problems in reinforcement learning: credit assignment. You also are suggesting a typical approach to address it: reward engineering or reward shaping. This is an open problem and there is no definitive answer to it. You could have a look at previous attempts, like this one: curiouscoder.space/blog/machine%20learning/… $\endgroup$ – ncasas Mar 11 at 21:58
1
$\begingroup$

It depends on the goal of the project. Projects can be a spectrum from "pure" to "applied":

A completely pure project only give minimal information. In the case of the Blackjack reward signal, that would just provide the win, loss, or tie information to the agent at the end of each hand.

A more applied project could give more information to the agent before, during, or after each hand.

An applied project will most likely start training better, learn faster, and end training higher. However, an applied project is less likely to provide evidence about the specific capacities of the system.

For both pure and applied projects, the amount and type of reward signal is a hyperparameter that is best tuned through empirical experimentation.

| improve this answer | |
$\endgroup$
0
$\begingroup$

If you include extra rules/'hints' like providing a reward for hitting on count <10, you're adding expert knowledge. With enough playouts, your expected rewards should be higher playing 'correctly.' Are you using eligibility traces in your algorithm?

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.