4
$\begingroup$

I am thinking of the following image where we have two weights and an error (so we can make a 3D visualization). In this case a "long valley" looks like $x^2$ in the plane of the gradient, but in a perpendicular plane the function can still be minimized:

enter image description here

According to my understanding, all optimization algorithms which are based on the gradient should be "trapped" at the saddle point like SGD is. Why does having a momentum term help in this case?

(I'm sorry, I don't know who made those images. More of them are available.)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The answer is simple: in a perfect situation never. These optimizers are 1st order methods (unlike for example Hessian-Free optimization). The reason for they escaping from the stationary point above is that there is some small gradient (too small for ordinary SGD) in the "correct" direction. Momentum and Nesterov gradient can build up the velocity over time in that direction and Adagrad/Adadelta/RMS can immediately scale the gradient (increase it in the direction for which the past gradients were small).

Improve this answer
$\endgroup$
0
$\begingroup$

The animation is by Alec Radford.

Momentum helps by "pushing through" the local minimum or saddle point. Of course, it does not push in the optimal direction, which is why it exhibits oscillation, as you can see above, but it eventually recovers.

For further study, I suggest reading about preconditioning and the condition number.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.