What is the bellman equation update rule for the average reward reinforcement learning? I searched a few articles, but could not find any practical answer.


In general, the average reward setting replaces the discounted setting in continuous tasks. It relies on there being a long term stable distribution of states under any particular policy (this is called ergodicity) - and in this will usually be true for continuous MDPs that don't have absorbing states.

If you see an update rule in the discounted setting that looks like this (for Q learning):

$$Q(s,a) \leftarrow Q(s,a) + \alpha(r + \gamma\text{max}_{a'}Q(s',a') - Q(s,a))$$

Then you replace the discounted TD error by the differential TD error:

$$Q(s,a) \leftarrow Q(s,a) + \alpha(r -\bar{r} + \text{max}_{a'}Q(s',a') - Q(s,a))$$

where $\bar{r}$ is the mean reward per time step under the current policy. You can estimate this simply from the rewards seen so far.

I searched a few articles, but could not find any practical answer.

See Reinforcement Learning: An Introduction (Second Edition) chapter 10, sections 3 and 4 for a more thorough description and more examples.

| improve this answer | |
  • $\begingroup$ Thank you again neil!.But I could not understand the intuition behind differential TD error how this equation carry average reward with thousands of states can you give me a simple example $\endgroup$ – uğur yıldırım Jul 7 '19 at 18:22
  • $\begingroup$ @uğuryıldırım: The average reward is not related to individual states. It is an average over all time steps. For more details and intuition, please read chapter 10 of Sutton & Barto. $\endgroup$ – Neil Slater Jul 7 '19 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.