3
$\begingroup$

I am looking into policy gradient methods. I stumbled into this implementation: https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a It uses a Nesterov Adam optimizer.

If I run it, it converges and gets good scores on OpenAI Gym's CartPole-v0.

However, if I change the optimizer from Adam to stochastic gradient descent (SGD), it never converges and seems to act randomly. Why is this? Is there something about policy gradient methods which make SGD a poor choice?

NOTE: there is a bug in that code which only runs the episode for 100 time steps. The episode can run for up to 200 time steps. I fixed this when running it.

Improve this question
$\endgroup$
11
  • $\begingroup$ What happens if you remove your 100 time step limit fix? I ask because you have to be careful with that, some Gym environments return the done flag to signal episode max time steps. They should not really return that, it can send spurious signals to your agent. One way to avoid the issue is to terminate earlier, and reset the environment, at the agent. $\endgroup$
    – Neil Slater
    Commented Sep 17, 2018 at 14:50
  • $\begingroup$ I am not sure why do you suggest that the done variable affects performance. I have used it with success in many of the OpenAI envs. I even include it in my custom envs. Apart from that its usage is needed especially if you are using Generalized Advantage Estimators or n-step rewards as it indicates when an episode is done so you dont propagate the collected reward from the next sequence of actions back to the first one. @Atuos: For SGD did you try to change the learning rate? $\endgroup$
    – Constantinos
    Commented Sep 18, 2018 at 5:42
  • 1
    $\begingroup$ @Constantinos: At least one gym environment - Lunar Lander, returns done to signal a timeout that is not part of the problem being solved. This is a major problem for environments which may end with a negative reward, such as LunarLander-v2, because ending the episode by timing out may be preferable to other solutions. The agent will learn to time out the episode instead of solving the problem presented. In some environments that might indeed be the goal, but in others it is spurious, and needs to be worked around or ignored. $\endgroup$
    – Neil Slater
    Commented Sep 23, 2018 at 9:06
  • 1
    $\begingroup$ @Constantinos: It's more fundamental than that. The problem is that done == "terminal state". It is the only way that gym flags a terminal state. This is not to do with quality of agent, but the definition of the problem being solved. If the timeout was actually part of the problem to be solved (i.e. "the agent has run out of time" in an episodic problem), then the time step id should be part of the state. It isn't in gym, and problems in Open AI's gym that timeout using done flag are not well-formed MDPs $\endgroup$
    – Neil Slater
    Commented Sep 24, 2018 at 7:01
  • 1
    $\begingroup$ @Constantinos: I am aware of that. My comments are about environments that do have terminal states. Gym's CartPolve-v0 is one of them, and in general Gym does not flag terminal states vs continuing-but-timed-out-pragmatically situations differently. That is a problem that needs handling in any agent code $\endgroup$
    – Neil Slater
    Commented Sep 27, 2018 at 6:15

2 Answers 2

Reset to default
1
$\begingroup$

Stochastic gradient descent (SGD) can get stuck in saddle points which results in the model not converging. A saddle points gradient is where zero in many directions but not all directions.

Adam adaptively changes the learning rate based on past gradients. This additional information can allow the model to escape saddle points, thus the model is more likely to converge.

Improve this answer
$\endgroup$
0
$\begingroup$
  • ADAM: Adaptive Moment Estimation, this optimization algorithm take both momentum and the sum squares of the gradient is considered when calculating delta for the next iteration.
  • SDG: Gradient Descent in TF-Keras can be implemented with momentum modification (not by default though), this is simply the previous gradient multiply by some decay rate. If no momentum is used then it is as Brian said before. If some momentum was used then the result may just be converging slower because SDG may take a longer path to get to the minimum without accounting for the sum squares. For more info, this is a great article: https://towardsdatascience.com/a-visual-explanation-of-gradient-descent-methods-momentum-adagrad-rmsprop-adam-f898b102325c
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.