Why in this case are gradient steps not perpendicular to contour lines?

Question

There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected?

source: http://www.deeplearningbook.org/contents/numerical.html page 89

And also in this case: source: http://www.deeplearningbook.org/contents/optimization.html page 310

In second case it is even weirder because the steps don't seem to be perpendicular to contour lines but to previous gradient vector? Am I missing something or treat images to serious? Thank in advance and regards.

Answer 1

That would be the case if the steps taken by gradient descent are "infinitesimal" in the mathematical sense. But in fact, it takes steps with some finite length defined by the learning rate. But the problem is, the gradient is calculated at each point, and does not really calculate how the slope changes at some finite length later. If you choose a large learning rate, you may "offshoot" from the optimal direction as shown in the figure. If you choose a small enough learning rate those oscillations will be minimal and you may move "almost" perpendicular to the contour lines and look like what you describe in the question, but it will take a very long time to complete training.

The second part of the question refers to the section titled "Conjugate Gradients" and it refers to a specific optimization method. The reason for the perpendicular lines in the second part is because of the vanishing gradient at the turning points. Quoting the text:

The method of steepest descent involves jumping to the point of lowest cost along the line defined by the gradient at the initial point on each step

Such a point is where the "directional derivative" with respect to theta, theta being the parameter in a given direction vanishes. When the derivative vanishes with respect to the parameter of the direction, The only possible way to go will be in a perpendicular direction. That is why they are called "conjugate" to each other, hence the name of the optimization method. This is detailed in the text above the figure.

To summarize the situation for that figure; An initial direction is chosen at the starting point by calculating the gradient there, and you keep moving in that direction until the gradient vanishes. That point is the minimum in that direction, so if you go any further on that straight line, the cost function will increase. The only direction you can go in which the cost function does not increase can be perpendicular to the initial direction.

To visualize this, look at image 4.5 in your first upload - page 88. Starting from that top of the saddle and following the gradient, you go down to the middle of the saddle. At that point going any further in the same direction will only take you up, while you want to go down. The only way to achieve this is to take a 90 degree turn at the middle of the saddle.