Questions tagged [momentum]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
0answers
45 views

Dissecting and understanding the Adam optimization's formula

Adam's optimization has the fololwing parameter update rule : $$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space ...
1
vote
0answers
23 views

Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
4
votes
1answer
766 views

Adam optimizer for projected gradient descent

The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc. The Adam optimizer is an improvement ...
4
votes
1answer
2k views

Why does momentum need learning rate?

If momentum optimizer independently keeps a custom "inertia" value for each weight, then why do we ever need to bother with learning rate? Surely, momentum would catch up its magnutude pretty ...