I am looking into policy gradient methods. I stumbled into this implementation: https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a It uses a Nesterov Adam optimizer.
If I run it, it converges and gets good scores on OpenAI Gym's CartPole-v0.
However, if I change the optimizer from Adam to stochastic gradient descent (SGD), it never converges and seems to act randomly. Why is this? Is there something about policy gradient methods which make SGD a poor choice?
NOTE: there is a bug in that code which only runs the episode for 100 time steps. The episode can run for up to 200 time steps. I fixed this when running it.