There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected? gradient descent

source: http://www.deeplearningbook.org/contents/numerical.html page 89

And also in this case: steepest descent source: http://www.deeplearningbook.org/contents/optimization.html page 310

In second case it is even weirder because the steps don't seem to be perpendicular to contour lines but to previous gradient vector? Am I missing something or treat images to serious? Thank in advance and regards.


1 Answer 1


That would be the case if the steps taken by gradient descent are "infinitesimal" in the mathematical sense. But in fact, it takes steps with some finite length defined by the learning rate. But the problem is, the gradient is calculated at each point, and does not really calculate how the slope changes at some finite length later. If you choose a large learning rate, you may "offshoot" from the optimal direction as shown in the figure. If you choose a small enough learning rate those oscillations will be minimal and you may move "almost" perpendicular to the contour lines and look like what you describe in the question, but it will take a very long time to complete training.

The second part of the question refers to the section titled "Conjugate Gradients" and it refers to a specific optimization method. The reason for the perpendicular lines in the second part is because of the vanishing gradient at the turning points. Quoting the text:

The method of steepest descent involves jumping to the point of lowest cost along the line defined by the gradient at the initial point on each step

Such a point is where the "directional derivative" with respect to theta, theta being the parameter in a given direction vanishes. When the derivative vanishes with respect to the parameter of the direction, The only possible way to go will be in a perpendicular direction. That is why they are called "conjugate" to each other, hence the name of the optimization method. This is detailed in the text above the figure.

To summarize the situation for that figure; An initial direction is chosen at the starting point by calculating the gradient there, and you keep moving in that direction until the gradient vanishes. That point is the minimum in that direction, so if you go any further on that straight line, the cost function will increase. The only direction you can go in which the cost function does not increase can be perpendicular to the initial direction.

To visualize this, look at image 4.5 in your first upload - page 88. Starting from that top of the saddle and following the gradient, you go down to the middle of the saddle. At that point going any further in the same direction will only take you up, while you want to go down. The only way to achieve this is to take a 90 degree turn at the middle of the saddle.

  • 1
    $\begingroup$ Thank you for the answer. But I believe that despite the fact steps are not "infinitesimal" the angle at the point should belong constant and should look more like this: imgur.com/a/NLn5hak Am I wrong? $\endgroup$
    – MajorTom
    Commented Nov 19, 2019 at 15:43
  • $\begingroup$ @MajorTom If your starting point is not on one of the lines I have drawn in this figure, the direction with the steepest descent will not point to the origin. This will also be the case after every epoch, you need to be on those lines to get to the center fastest. Any other point will make a zig-zag pattern, as the perpendicular to the ellipse there will not point towards origin. imgur.com/a/wDrTOZR – serali 8 mins ago Delete $\endgroup$
    – serali
    Commented Nov 19, 2019 at 16:12
  • $\begingroup$ @serali Yes, the starting points in OP's diagram are not on any printed contour line. But if you actually tried to draw a contour line below each starting point, you would see that they had to be very wavy to be orthogonal. This waviness contradicts the caption claiming the function to be quadratic. ¶ Hence, I think the diagram is a confusing example for gradient descent, because it is very inaccurate. $\endgroup$
    – Socowi
    Commented Feb 17, 2022 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.