10
$\begingroup$

It seems the Adaptive Moment Estimation (Adam) optimizer nearly always works better (faster and more reliably reaching a global minimum) when minimising the cost function in training neural nets.

Why not always use Adam? Why even bother using RMSProp or momentum optimizers?

$\endgroup$
  • 1
    $\begingroup$ I don't believe there is any strict, formalized way to support either statement. It's all purely empirical, as error surface is unknown. As a rule of thumb, and purely from m experience, ADAM does well where others fail (instance segmentation), although not without drawbacks (convergence is not monotone) $\endgroup$ – Alex May 8 '18 at 8:53
  • $\begingroup$ Adam is faster to converge. SGD is slower but generalizes better. So at the end it all depends on your particular circumstances. $\endgroup$ – agcala Mar 21 at 12:10
10
$\begingroup$

Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM. https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/

There is often a value to using more than one method (an ensemble), because every method has a weakness.

$\endgroup$
3
$\begingroup$

You should also take a look at this post comparing different gradient descent optimizers. As you can see below Adam is clearly not the best optimizer for some tasks as many converge better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.