50
$\begingroup$

It seems the Adaptive Moment Estimation (Adam) optimizer nearly always works better (faster and more reliably reaching a global minimum) when minimising the cost function in training neural nets.

Why not always use Adam? Why even bother using RMSProp or momentum optimizers?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ I don't believe there is any strict, formalized way to support either statement. It's all purely empirical, as error surface is unknown. As a rule of thumb, and purely from m experience, ADAM does well where others fail (instance segmentation), although not without drawbacks (convergence is not monotone) $\endgroup$
    – Alex
    May 8, 2018 at 8:53
  • 6
    $\begingroup$ Adam is faster to converge. SGD is slower but generalizes better. So at the end it all depends on your particular circumstances. $\endgroup$
    – agcala
    Mar 21, 2019 at 12:10
  • $\begingroup$ en.wikipedia.org/wiki/No_free_lunch_theorem would seem relevant. Different optimization algorithms work better on different problems, and there is no universally superior one. $\endgroup$
    – endolith
    Nov 28, 2022 at 20:55

2 Answers 2

Reset to default
40
$\begingroup$

Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM.

There is often a value to using more than one method (an ensemble), because every method has a weakness.

Improve this answer
$\endgroup$
18
$\begingroup$

You should also take a look at this post comparing different gradient descent optimizers. As you can see below Adam is clearly not the best optimizer for some tasks as many converge better.

Improve this answer
$\endgroup$
4
  • 4
    $\begingroup$ Just for the record: In the linked article they mention some of the flaws of ADAM and present AMSGrad as a solution. However, they conclude that whether AMSGrad outperform ADAM in practices is (at the time of writing) non-conclusive. $\endgroup$
    – Lus
    Sep 19, 2019 at 11:24
  • $\begingroup$ this link appears to be offline $\endgroup$
    – OuttaSpaceTime
    Jul 26, 2022 at 18:17
  • $\begingroup$ @OuttaSpaceTime the link works for me! (not sure if fixed or edited) $\endgroup$
    – tturbo
    Nov 4, 2022 at 10:11
  • 1
    $\begingroup$ @tturbo you are right! now it's working. Thanks for pointing this out $\endgroup$
    – OuttaSpaceTime
    Nov 4, 2022 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.