I'm new to reinforcement learning. I'm studying DQN with decaying epsilon. I came across such example:


GAMMA = 0.2




My questions are:

  1. Is it correct if epsilon doesn't reach MIN_EPSILON?
  2. Is there something wrong with the reward - the reward is not higher and higher but it is behaving otherwise - it decreases in time?


Average reward


1 Answer 1

  1. If you set epsilon decay to 0.999 you will need $$ \epsilon_{max} \cdot \epsilon_{decay}^x = \epsilon_{min} \\ 1 \cdot 0.999^x = 0.01 \\ x \approx 4603 $$ 4603 episodes to reach minimum epsilon. After 91 episodes you will reach $$ \epsilon_{current} = \epsilon_{max} \cdot \epsilon_{decay}^{episodes} = 1 \cdot 0.999^{91} \approx 0.913 $$ which is exactly what you can see in your plot. It's not a problem but remember that this model still makes over 91% moves randomly.
  2. Average reward should not decrease over time. It can mean a few things for example error in dqn algorithm or too high learning rate in your model. The best way to debug is to start with as simple environment as possible and let your model learn to play it and only then increase the difficulty.
  • $\begingroup$ So what is the sense of using GAMMA in such example if epsilon is for randomness of actions? It is for exploration-exploitation problem and set here to 0.2 means that 20% of actions are taken randomly? $\endgroup$
    – Martin
    Sep 9, 2020 at 11:18
  • $\begingroup$ Epsilon is the parameter which control how often you explore paths which the model considers inferior.Gamma is the parameter which controls how valuable the reward in the future is compared to reward you receive in the next step. In other words if your gamma is equal to 0.2 and the model thinks you will get reward equal 1 in the first step and -2 in the second step and 3 in the 3rd step it multiplies next steps by gamma so predicted value would be 1+gamma*(-2) + gamma^2 * (3). $\endgroup$
    – YuseqYaseq
    Sep 9, 2020 at 11:21
  • $\begingroup$ The way you showed to calculate number of epochs is for log decaying? It is applied always or is there something as linear decaying in such problems? $\endgroup$
    – Martin
    Sep 9, 2020 at 12:02
  • $\begingroup$ You can decay epsilon linearly but it will decay much more quickly which usually isn't desirable. If you decay epsilon too quickly your model will have a hard time learning new, unknown states which may increase time needed for convergence. I haven't seen linear epsilon decay used anywhere myself. $\endgroup$
    – YuseqYaseq
    Sep 9, 2020 at 12:31
  • $\begingroup$ Thanks a lot for your help! :) $\endgroup$
    – Martin
    Sep 9, 2020 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.