2
$\begingroup$

I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. I initialize the model with a pre-trained RNN model (a.k.a. warm start). This pre-trained model (trained with log-likelihood objective) got 0.6 F1 score in my task.

When I use adam optimizer to train this policy gradient objective, the performance of my model drops to 0 after a few epochs. However, if I switch to gradientdescent optimizer and keep everything else the same, the performance looks reasonable and slightly better than the pre-trained model. Is there any idea why is that?

I use tensorflow to implement my model.

Improve this question
$\endgroup$
2
  • $\begingroup$ maybe you could show the relevant pieces of code otherwise people have to guess what could be wrong :) $\endgroup$
    – oW_
    Commented Apr 10, 2019 at 20:58
  • $\begingroup$ Thank you! I don't think there is any bug in my code, because I tested it so many times with different inputs. I just want to know if in theory there is any idea of why this happened. $\endgroup$
    – Kechen
    Commented Apr 11, 2019 at 20:11

1 Answer 1

Reset to default
0
$\begingroup$

Without the code there's not much we can do but, I'd guess you need to significantly lower the learning rate. From my experience Adam requires a significantly lower learning rate compared to SGD.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I do start from a very small learning rate, but it does not help. I am wondering if adam failed because of its momentum mechanism. In RL sequence training, it is very likely to sample a very bad sequence which has 0 reward and the model get 0 gradient with that sample. However, adam with momentum still updates the model even if the gradient is zero. Would that be a problem? $\endgroup$
    – Kechen
    Commented Apr 11, 2019 at 20:14
  • $\begingroup$ @Kechen But that shouldn't make the model diverge, It should only make it learn slower $\endgroup$
    – Ran Elgiser
    Commented Apr 11, 2019 at 20:37
  • $\begingroup$ so you think the problem is the implementation? Do you have any experience on training RNN with RL objective? $\endgroup$
    – Kechen
    Commented Apr 11, 2019 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.