Why does decreasing the SGD learning rate cause a massive increase in accuracy?

Question

In papers such as this I often see training curves with this kind of shape:

In this case SGD was used with a factor of 0.9 and learning rate decreasing by a factor of 10 every 30 epochs.

• Why is there such a large decrease in error when the learning rate is changed?
• Why does the validation error begin to increase after the initial drop, whereas the training error continues decreasing?
• Can the same results be obtained by moving the 2nd and subsequent learning rate changes closer together? That is, why the delay in doing further drops?

Answer 1

With a higher learning rate, you take bigger steps towards the solution. However, when you are close, you might jump over the solution and then the next step, you jump over it again causing an oscillation around the solution. Now, if you lower the learning rate correctly, you will stop the oscillation and continue towards the solution once again. That is, until you start oscillating again. To keep in mind is that a larger learning rate can jump over smaller local minima and help you find a better minima, which it can't jump over. Also, it is generally the training error that becomes better, and the validation error becomes worse as you start to overfit on the training data.

Answer 2

Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting. The validation error oscillated approaching the second saddle point. The noise makes it difficult to state that it increased with statistical significance, but if it did it could be due to overfitting. I do not know of any result that relates the separation between saddle points, so the delay could be arbitrary. At some point you reach the bottom, of course.