6
$\begingroup$

How to select between average reward and discounted reward?

  • And when average reward is more effective in comparison with discounter reward and when vice versa is correct?

  • Is is possible to use both of them in a problem? Because as I understand the RL reward is based on average reward or discounted future reward, but I think this paper use the discounted and average together. Is it correct: we use discounted future reward in order to training and average reward in test and evaluation? What is wrong in my understanding?

In this picture, figure 2 of the paper "Playing Atari with Deep Reinforcement Learning":

enter image description here

The authors report the "average reward". However, in the same paper, the authors also mention "discounted reward". So, I'm confused. What is the difference between discounted reward and average reward?

$\endgroup$
1
  • $\begingroup$ Sorry I edit my question the maximization of discounted reward is calculated but for comparison the average reward is considered, while I know average reward and discounted future reward as two methods for comparison performace of RL! What is wrong in my understanding? $\endgroup$ Feb 19, 2019 at 4:25

2 Answers 2

1
$\begingroup$

The Average reward in that figure is used as a measure of performance. In other words, the score of the agent playing the game. You do not track reward per episode as this doesnt indicate a general improvement in the learning process. Instead you track the average reward over training epochs. If it steadily increases this means that your agent indeed is learning.

The discounted reward is used to create some kind of future-reward dependencies and is used in the learning equations. So, instead of evaluating how good is a particular state according to the immediate reward you received, you also take into account the future reward from your next state. In RL you attempt to max your expected return and some methods estimate the expected reward from every state.

Please note that my answer gives a high level description and is not referring to a specific RL algorithm (as there are many variations). I would suggest you to understand very well the simple tabular form of Q-learning before moving to RL and function approximators combinations.

$\endgroup$
5
  • $\begingroup$ Thanks but I think this is not my answer. I asked that in this paper, the maximization of discounted reward is calculated but for comparison the average reward is considered, while I know average reward and discounted future reward as two methods for comparison performace of RL! What is wrong in my understanding? $\endgroup$ Feb 19, 2019 at 4:24
  • $\begingroup$ The discounted reward is NOT used for performance comparison (in general). It is only used in the learning equations (for theoretical reasons - for derivations you can refer to the classic RL book by Sutton and Barto). The average (immediate) reward per time frame is used for performance comparison. $\endgroup$ Feb 19, 2019 at 17:30
  • $\begingroup$ OK, thanks what is difference between two right and left? $\endgroup$ Feb 20, 2019 at 0:49
  • $\begingroup$ From what I read, the authors used the max Q because the score seemed fluctuating a lot. By looking at the increasing Q we can conclude that the algorithm is improving across episodes. Sometimes, during training, will be equal to the max score/episode indicating that your algorithm has converged. If you want to compare different algorithms use the score. If you want insights of what your agent is estimating during an episode look at the reward predictions (value functions). The score is just the immediate reward whilst the values contain agent's estimation about reward expectations. $\endgroup$ Feb 20, 2019 at 2:44
  • $\begingroup$ "Sometimes, during training, will be equal to the max score/episode" depending on the task and its reward function. The score is a general objective measure of performance for different algorithms. You will notice that in all papers that compare different RL algorithms, its the score that is mentioned. $\endgroup$ Feb 20, 2019 at 5:18
1
$\begingroup$

In order to consider they can be used together or not, let's see it this way.

Discounting is determined by the "discounting factor" or gamma symbol in the paper. This hyper-parameter always exists in the calculation of return. You can adjust your environment to have no discount at all by setting gamma=1, or you can choose to have the discount by setting gamma below 1. Adjusting this is likely to affect your learning performance.

For the average reward in the figure2, the paper says "One epoch corresponds to 50000 minibatch weight updates or roughly 30 minutes of training time". During this 30 minutes, the agent will not play for just one episode, but a lot of them. Each played episode generates a return (the total reward which contains that gamma in the calculation). The average reward is calculated directly from those episodes in the same epoch. If you don't like average, you may choose max or min or any other operator. It depends on what you want to see.

These two things are two knobs that you can choose to adjust independently.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.