Q learning for blackjack, reward function?

Question

I am currently learning reinforcement learning and am have built a blackjack game.

There is an obvious reward at the end of the game (payout), however some actions do not directly lead to rewards (hitting on a count of 5), which should be encouraged, even if the end result is negative (loosing the hand).

My question is what should the reward be for those actions ?

I could hard code a positive reward (fraction of the reward for winning the hand) for hits which do not lead to busting, but it feels like I am not approaching the problem correctly.

Also, when I assign a reward for a win (after the hand is over), I update the q-value corresponding to the last action/state pair, which seems suboptimal, as this action may not have directly lead to the win.

Another option I thought is to assign the same end reward to all of the action/state pairs in the sequence, however, some actions (like hitting on count <10) should be encouraged even if it leads to a lost hand.

Note: My end goal is to use deep-RL with an LSTM, but I am starting with q-learning.

Answer 1

It depends on the goal of the project. Projects can be a spectrum from "pure" to "applied":

A completely pure project only give minimal information. In the case of the Blackjack reward signal, that would just provide the win, loss, or tie information to the agent at the end of each hand.

A more applied project could give more information to the agent before, during, or after each hand.

An applied project will most likely start training better, learn faster, and end training higher. However, an applied project is less likely to provide evidence about the specific capacities of the system.

For both pure and applied projects, the amount and type of reward signal is a hyperparameter that is best tuned through empirical experimentation.

Answer 2

If you include extra rules/'hints' like providing a reward for hitting on count <10, you're adding expert knowledge. With enough playouts, your expected rewards should be higher playing 'correctly.' Are you using eligibility traces in your algorithm?