Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

# Questions tagged [momentum]

The tag has no usage guidance.

8 questions
Filter by
Sorted by
Tagged with
11 views

### How do you find the eigenvalues of the matrix for the following momentum gradient descent?

The following question is based purely on the material available on MIT's open courseware youtube channel. (https://www.youtube.com/watch?v=wrEcHhoJxjM). In it, Professor Gilbert Strang explains the ...
22 views

### Is the usage of the "momentum" significiantly superior to the conventional weight update

The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller). Is it significiantly ...
59 views

### Adam Optimiser First Step

Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (...
• 123
15k views

### What is momentum in neural network?

While using "Two class neural network" in Azure ML, I encountered "Momentum" property. As per documentation, which is not clear, it says For The momentum, type a value to apply ...
74 views

### Dissecting and understanding the Adam optimization's formula

Adam's optimization has the fololwing parameter update rule : $$\theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where  m_t \text{ is first moment of gradients and} \space ...
1 vote
34 views

### Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
1k views

### Adam optimizer for projected gradient descent

The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc. The Adam optimizer is an improvement ...
• 2,968