Questions tagged [momentum]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
20 views

Is the usage of the “momentum” significiantly superior to the conventional weight update

The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller). Is it significiantly ...
2
votes
1answer
37 views

Adam Optimiser First Step

Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (...
2
votes
0answers
61 views

Dissecting and understanding the Adam optimization's formula

Adam's optimization has the fololwing parameter update rule : $$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space ...
1
vote
0answers
26 views

Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
5
votes
2answers
942 views

Adam optimizer for projected gradient descent

The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc. The Adam optimizer is an improvement ...
4
votes
1answer
2k views

Why does momentum need learning rate?

If momentum optimizer independently keeps a custom "inertia" value for each weight, then why do we ever need to bother with learning rate? Surely, momentum would catch up its magnutude pretty ...