I am looking into policy gradient methods. I stumbled into this implementation: https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a It uses a Nesterov Adam optimizer.

If I run it, it converges and gets good scores on OpenAI Gym's CartPole-v0.

However, if I change the optimizer from Adam to stochastic gradient descent (SGD), it never converges and seems to act randomly. Why is this? Is there something about policy gradient methods which make SGD a poor choice?

NOTE: there is a bug in that code which only runs the episode for 100 time steps. The episode can run for up to 200 time steps. I fixed this when running it.

  • $\begingroup$ What happens if you remove your 100 time step limit fix? I ask because you have to be careful with that, some Gym environments return the done flag to signal episode max time steps. They should not really return that, it can send spurious signals to your agent. One way to avoid the issue is to terminate earlier, and reset the environment, at the agent. $\endgroup$ – Neil Slater Sep 17 '18 at 14:50
  • $\begingroup$ I am not sure why do you suggest that the done variable affects performance. I have used it with success in many of the OpenAI envs. I even include it in my custom envs. Apart from that its usage is needed especially if you are using Generalized Advantage Estimators or n-step rewards as it indicates when an episode is done so you dont propagate the collected reward from the next sequence of actions back to the first one. @Atuos: For SGD did you try to change the learning rate? $\endgroup$ – Constantinos Sep 18 '18 at 5:42
  • $\begingroup$ @Constantinos: At least one gym environment - Lunar Lander, returns done to signal a timeout that is not part of the problem being solved. This is a major problem for environments which may end with a negative reward, such as LunarLander-v2, because ending the episode by timing out may be preferable to other solutions. The agent will learn to time out the episode instead of solving the problem presented. In some environments that might indeed be the goal, but in others it is spurious, and needs to be worked around or ignored. $\endgroup$ – Neil Slater Sep 23 '18 at 9:06
  • $\begingroup$ @Neil done always indicates the end of your trials and has nothing to do with the problem being solved or not. You need only this and then use the trajectory generated so you can do PG, DQN, etc. If the agent chooses to loose (i.e ending the episode sooner) and never experiences the final reward then the agent is not suitable for that env (or you need some reward shaping). Imagine Montezuma revenge using simple DQN and getting penalty for every step. The agent of course will prefer to loose as soon as possible. But this means that the agent is not suitable and needs a more sophisticated one. $\endgroup$ – Constantinos Sep 24 '18 at 0:53
  • $\begingroup$ @Constantinos: It's more fundamental than that. The problem is that done == "terminal state". It is the only way that gym flags a terminal state. This is not to do with quality of agent, but the definition of the problem being solved. If the timeout was actually part of the problem to be solved (i.e. "the agent has run out of time" in an episodic problem), then the time step id should be part of the state. It isn't in gym, and problems in Open AI's gym that timeout using done flag are not well-formed MDPs $\endgroup$ – Neil Slater Sep 24 '18 at 7:01

Stochastic gradient descent (SGD) can get stuck in saddle points which results in the model not converging. A saddle points gradient is where zero in many directions but not all directions.

Adam adaptively changes the learning rate based on past gradients. This additional information can allow the model to escape saddle points, thus the model is more likely to converge.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.