Discounted rewards seems unbalanced to me.

If we take as example an episode with 4 actions, where each action receive a reward of +1 :

+1 -> +1 -> +1 -> +1

The discounted reward for the last action is : 1
The discounted reward for the first action (considering gamma = 1 for simplicity) is : 4

Intuitively both action are as good as the other, because both received same reward.
But their total reward is different, unbalanced.

So when we will backpropagate, first action will be favored over last action ?


1 Answer 1


Most of the RL problems are sequential decision making processes. That is the action taken at time $t$, can influence the future rewards in a sequential way. From this perspective, the first action lead to a sequence of states/actions which produced positive rewards, which means it deserves more positive feedback than the last action. It only seems natural to prefer the first action in the first state unless we know some other action yields better cumulative rewards. Rewards are gains in short-term, an action might be good in the short term but not in the long run, which is why we use cumulative rewards called returns to estimate the value of an action in a given state.

  • $\begingroup$ It makes sense ! So we agree that in the case when the actions are not a sequence (i.e. first action does not influence the other actions), discount does not make sense ? $\endgroup$
    – Astariul
    Commented Aug 8, 2019 at 6:33
  • 1
    $\begingroup$ Yes, it doesn't make sense when there is no sequential reaction. $\endgroup$ Commented Aug 8, 2019 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.