3
$\begingroup$

I'm a beginner in Machine learning and i'm searching for some optimizer for the gradient descent. I've searched many topics about that, and did a state of art of all these optimizers. I have just one problem, and i can't figure it out. Don't judge me please, but i would like to know ?

Are we using ADAM optimizer alone or are we obliged to combine it with the SGD ? I didn't understand if it works alone or if it's here to optimize NOT the neural network but the SGD of the neural network?

Thank you for your help!

$\endgroup$
2
  • 2
    $\begingroup$ ADAM is a standalone optimizer, so no need to be combined with SGD. It is kind of advanced flavor of classical SGD. Have you seen this post: machinelearningmastery.com/…? I think it has all you need to know about ADAM, and it is even compared with others. $\endgroup$ Aug 12, 2018 at 18:25
  • $\begingroup$ Yep, as an aside, just about any optimizer you will come across is still a form of SGD, just with fancier handling of different weights and learning rate over time $\endgroup$
    – Sean Owen
    Nov 29, 2021 at 15:22

1 Answer 1

1
$\begingroup$

Adam optimization is an extension of stochastic gradient descent (SGD) optimization.

SGD maintains a single learning rate for all weight updates and the learning rate does not change during training.

Adam optimization can have a different learning rate for each weight and change the learning rate during training.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.