I am thinking of the following image where we have two weights and an error (so we can make a 3D visualization). In this case a "long valley" looks like $x^2$ in the plane of the gradient, but in a perpendicular plane the function can still be minimized:
According to my understanding, all optimization algorithms which are based on the gradient should be "trapped" at the saddle point like SGD is. Why does having a momentum term help in this case?
(I'm sorry, I don't know who made those images. More of them are available.)