In papers such as this I often see training curves with this kind of shape:

enter image description here

In this case SGD was used with a factor of 0.9 and learning rate decreasing by a factor of 10 every 30 epochs.

  • Why is there such a large decrease in error when the learning rate is changed?
  • Why does the validation error begin to increase after the initial drop, whereas the training error continues decreasing?
  • Can the same results be obtained by moving the 2nd and subsequent learning rate changes closer together? That is, why the delay in doing further drops?

2 Answers 2


With a higher learning rate, you take bigger steps towards the solution. However, when you are close, you might jump over the solution and then the next step, you jump over it again causing an oscillation around the solution. Now, if you lower the learning rate correctly, you will stop the oscillation and continue towards the solution once again. That is, until you start oscillating again. To keep in mind is that a larger learning rate can jump over smaller local minima and help you find a better minima, which it can't jump over. Also, it is generally the training error that becomes better, and the validation error becomes worse as you start to overfit on the training data.


Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting. The validation error oscillated approaching the second saddle point. The noise makes it difficult to state that it increased with statistical significance, but if it did it could be due to overfitting. I do not know of any result that relates the separation between saddle points, so the delay could be arbitrary. At some point you reach the bottom, of course.

  • $\begingroup$ Sorry, do you mean larger learning allow escaping saddle points? This is also what @Carl in the other answer talks about? $\endgroup$
    – SmallChess
    Sep 23, 2017 at 1:04
  • $\begingroup$ No, smaller. Same subject. Imagine that the manifold that connects one local minima to another is through a narrow hole. You are unlikely to go through it if you take big steps. $\endgroup$
    – Emre
    Sep 23, 2017 at 2:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.