I am training a reinforcement learning agent on an episodic task of fixed episode length. I am tracking the training process by plotting the cumulative rewards over an episode. I am using tensorboard for plotting the rewards. I have trained my agent for 20M steps. So I believe the agent has been given enough time to train. The cumulative rewards for an episode can range from +132 to around -60. My plot with a smoothing of 0.999

enter image description here

Over the episodes, I can see that my rewards have converged. But if I see the plot with smoothing of 0

enter image description here

There is a huge variation in the rewards. So should I consider that the agent has converged or not? Also I don't understand why is there such a huge variation in rewards even after so much of training?


  • $\begingroup$ Are you using an off-policy method? You have tagged with DQN and Actor-Critic and Monte-Carlo - are you using any of these? It may be better to just tag with the one you are using. $\endgroup$ – Neil Slater Nov 29 '19 at 10:33
  • $\begingroup$ Edited the tags, I am using policy gradient methods(PPO and actor-critic) $\endgroup$ – cvg Nov 29 '19 at 10:37
  • $\begingroup$ OK, so that's on-policy I think, and a policy gradient method with on-policy should perform less exploration as it progresses. So it is probably not excess exploration confusing the signal. In which case I am not sure I can write an answer (I am also not very familiar with PPO) $\endgroup$ – Neil Slater Nov 29 '19 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.