53
$\begingroup$

It seems the Adaptive Moment Estimation (Adam) optimizer nearly always works better (faster and more reliably reaching a global minimum) when minimising the cost function in training neural nets.

Why not always use Adam? Why even bother using RMSProp or momentum optimizers?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ I don't believe there is any strict, formalized way to support either statement. It's all purely empirical, as error surface is unknown. As a rule of thumb, and purely from m experience, ADAM does well where others fail (instance segmentation), although not without drawbacks (convergence is not monotone) $\endgroup$
    – Alex
    Commented May 8, 2018 at 8:53
  • 6
    $\begingroup$ Adam is faster to converge. SGD is slower but generalizes better. So at the end it all depends on your particular circumstances. $\endgroup$
    – agcala
    Commented Mar 21, 2019 at 12:10
  • $\begingroup$ en.wikipedia.org/wiki/No_free_lunch_theorem would seem relevant. Different optimization algorithms work better on different problems, and there is no universally superior one. $\endgroup$
    – endolith
    Commented Nov 28, 2022 at 20:55

2 Answers 2

Reset to default
41
$\begingroup$

Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM.

There is often a value to using more than one method (an ensemble), because every method has a weakness.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This post needs an update. It's not working at all. $\endgroup$
    – MasterOne Piece
    Commented Aug 28 at 18:54
18
$\begingroup$

You should also take a look at this post comparing different gradient descent optimizers. As you can see below Adam is clearly not the best optimizer for some tasks as many converge better.

Improve this answer
$\endgroup$
5
  • 4
    $\begingroup$ Just for the record: In the linked article they mention some of the flaws of ADAM and present AMSGrad as a solution. However, they conclude that whether AMSGrad outperform ADAM in practices is (at the time of writing) non-conclusive. $\endgroup$
    – Lus
    Commented Sep 19, 2019 at 11:24
  • 1
    $\begingroup$ this link appears to be offline $\endgroup$
    – OuttaSpaceTime
    Commented Jul 26, 2022 at 18:17
  • $\begingroup$ @OuttaSpaceTime the link works for me! (not sure if fixed or edited) $\endgroup$
    – tturbo
    Commented Nov 4, 2022 at 10:11
  • 1
    $\begingroup$ @tturbo you are right! now it's working. Thanks for pointing this out $\endgroup$
    – OuttaSpaceTime
    Commented Nov 4, 2022 at 11:21
  • 1
    $\begingroup$ The provided link is dead. $\endgroup$
    – JoyfulPanda
    Commented Jun 8 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.