7
$\begingroup$

I am looking for stabilizing my results of DQN, I found clipping is one technique to do it but I did not understand it completely!

1- what are the effects of clipping the reward, clipping the gradient, clipping the error in stability and how makes results more stable?

2- In DQN nature it has written they clipping the reward? Would you please explain this more?

3- which of them are more effective in stability?

$\endgroup$
2
  • $\begingroup$ what does the first question ask about? seems like the question misses a word?! also this is a bit broad. try to be more specific or ask multiple separate questions to get better/more feedback $\endgroup$
    – oW_
    Oct 1, 2018 at 20:43
  • $\begingroup$ I edit it somehow. Actually, I want to implement clipping the reward based on what is written in DeepMind paper. I think I have written my explanation by partitioning my question into several questions! Which section is still vague and broad? $\endgroup$ Oct 2, 2018 at 1:58

1 Answer 1

6
$\begingroup$

You could clip for several reasons.

  • If you clip the gradient, the stabilizing effect is to force the optimizer to do only small changes in the backward step. Of course, you could also decrease the learning rate but the effect is slightly different. When you decrease the learning rate you basically say "learn more slowly". Instead, with gradient clipping, you say "learn as usual but if you have to change your mind rapidly don't do it" (I'm not sure that this sentence is understandable, English isn't my first language).
  • If you clip the error, the effect is the same. Maybe it changes a bit in a mathematical point of view but the bigger result is equal to clipping the gradient.
  • Clipping the reward doesn't give you any direct stabilizing effect. It is only a particular case of the more general reward shaping. In Playing Atari with Deep Reinforcement Learning, it's stated that:

    Since the scale of scores varies greatly from game to game, we fixed all positive rewards to be 1 and all negative rewards to be −1, leaving 0 rewards unchanged. Clipping the rewards in this manner limits the scale of the error derivatives and makes it easier to use the same learning rate across multiple games. At the same time, it could affect the performance of our agent since it cannot differentiate between rewards of different magnitude.

    Basically, they've done it to make the environments similar to each other in rewards terms when seen by an agent. If you think of it, it's not difficult for you to play to a game when the score is given as multiple of thousand and then play to another one where even one hundred could be a great score. This doesn't apply to RL agents and so they reshaped the reward.

$\endgroup$
2
  • 1
    $\begingroup$ > This doesn't apply to RL agents and so they reshaped the reward Nice answer but: Why not? It's job is to maximize the accumulated reward, so why it makes a difference if it is a "multiple of thousand" or "one hundred"? $\endgroup$
    – guest
    Mar 15, 2020 at 3:24
  • 2
    $\begingroup$ The authors wanted to apply the same model to multiple games and each game had arbitrary points systems, in different scales. In order to mitigate the effects of rewards if different scales when using the same model, they just clipped everything to be between -1 and 1, IIRC. That doesn't mean a model can't benefit from the information provided by unclipped reward signals, but in Atari the reward is not very informative as it is abstract, so the authors didn't think it would be a problem to clip it. $\endgroup$ May 1, 2020 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.