I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. I initialize the model with a pre-trained RNN model (a.k.a. warm start). This pre-trained model (trained with log-likelihood objective) got 0.6 F1 score in my task.

When I use adam optimizer to train this policy gradient objective, the performance of my model drops to 0 after a few epochs. However, if I switch to gradientdescent optimizer and keep everything else the same, the performance looks reasonable and slightly better than the pre-trained model. Is there any idea why is that?

I use tensorflow to implement my model.

  • $\begingroup$ maybe you could show the relevant pieces of code otherwise people have to guess what could be wrong :) $\endgroup$ – oW_ Apr 10 '19 at 20:58
  • $\begingroup$ Thank you! I don't think there is any bug in my code, because I tested it so many times with different inputs. I just want to know if in theory there is any idea of why this happened. $\endgroup$ – Kechen Apr 11 '19 at 20:11

Without the code there's not much we can do but, I'd guess you need to significantly lower the learning rate. From my experience Adam requires a significantly lower learning rate compared to SGD.

  • $\begingroup$ I do start from a very small learning rate, but it does not help. I am wondering if adam failed because of its momentum mechanism. In RL sequence training, it is very likely to sample a very bad sequence which has 0 reward and the model get 0 gradient with that sample. However, adam with momentum still updates the model even if the gradient is zero. Would that be a problem? $\endgroup$ – Kechen Apr 11 '19 at 20:14
  • $\begingroup$ @Kechen But that shouldn't make the model diverge, It should only make it learn slower $\endgroup$ – Ran Elgiser Apr 11 '19 at 20:37
  • $\begingroup$ so you think the problem is the implementation? Do you have any experience on training RNN with RL objective? $\endgroup$ – Kechen Apr 11 '19 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.