It seems the Adaptive Moment Estimation (Adam) optimizer nearly always works better (faster and more reliably reaching a global minimum) when minimising the cost function in training neural nets.

Why not always use Adam? Why even bother using RMSProp or momentum optimizers?

  • 2
    $\begingroup$ I don't believe there is any strict, formalized way to support either statement. It's all purely empirical, as error surface is unknown. As a rule of thumb, and purely from m experience, ADAM does well where others fail (instance segmentation), although not without drawbacks (convergence is not monotone) $\endgroup$
    – Alex
    May 8, 2018 at 8:53
  • 5
    $\begingroup$ Adam is faster to converge. SGD is slower but generalizes better. So at the end it all depends on your particular circumstances. $\endgroup$
    – agcala
    Mar 21, 2019 at 12:10

2 Answers 2


Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM.

There is often a value to using more than one method (an ensemble), because every method has a weakness.


You should also take a look at this post comparing different gradient descent optimizers. As you can see below Adam is clearly not the best optimizer for some tasks as many converge better.

  • 4
    $\begingroup$ Just for the record: In the linked article they mention some of the flaws of ADAM and present AMSGrad as a solution. However, they conclude that whether AMSGrad outperform ADAM in practices is (at the time of writing) non-conclusive. $\endgroup$
    – Lus
    Sep 19, 2019 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.