I am looking into policy gradient methods. I stumbled into this implementation: https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a It uses a Nesterov Adam optimizer.

If I run it, it converges and gets good scores on OpenAI Gym's CartPole-v0.

However, if I change the optimizer from Adam to stochastic gradient descent (SGD), it never converges and seems to act randomly. Why is this? Is there something about policy gradient methods which make SGD a poor choice?

NOTE: there is a bug in that code which only runs the episode for 100 time steps. The episode can run for up to 200 time steps. I fixed this when running it.

• What happens if you remove your 100 time step limit fix? I ask because you have to be careful with that, some Gym environments return the done flag to signal episode max time steps. They should not really return that, it can send spurious signals to your agent. One way to avoid the issue is to terminate earlier, and reset the environment, at the agent. Sep 17 '18 at 14:50
• I am not sure why do you suggest that the done variable affects performance. I have used it with success in many of the OpenAI envs. I even include it in my custom envs. Apart from that its usage is needed especially if you are using Generalized Advantage Estimators or n-step rewards as it indicates when an episode is done so you dont propagate the collected reward from the next sequence of actions back to the first one. @Atuos: For SGD did you try to change the learning rate? Sep 18 '18 at 5:42
• @Constantinos: At least one gym environment - Lunar Lander, returns done to signal a timeout that is not part of the problem being solved. This is a major problem for environments which may end with a negative reward, such as LunarLander-v2, because ending the episode by timing out may be preferable to other solutions. The agent will learn to time out the episode instead of solving the problem presented. In some environments that might indeed be the goal, but in others it is spurious, and needs to be worked around or ignored. Sep 23 '18 at 9:06
• @Constantinos: It's more fundamental than that. The problem is that done == "terminal state". It is the only way that gym flags a terminal state. This is not to do with quality of agent, but the definition of the problem being solved. If the timeout was actually part of the problem to be solved (i.e. "the agent has run out of time" in an episodic problem), then the time step id should be part of the state. It isn't in gym, and problems in Open AI's gym that timeout using done flag are not well-formed MDPs Sep 24 '18 at 7:01
• @Constantinos: I am aware of that. My comments are about environments that do have terminal states. Gym's CartPolve-v0 is one of them, and in general Gym does not flag terminal states vs continuing-but-timed-out-pragmatically situations differently. That is a problem that needs handling in any agent code Sep 27 '18 at 6:15