Return to Answer

Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting. The validation error in the SE algorithm oscillated approaching the second saddle point. The noise makes it difficult to state that it increased with statistical significance, but if it did it could be due to overfitting. I do not know of any result that relates the separation between saddle points, so the delay could be arbitrary. At some point you reach the bottom, of course.

Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting. The validation error oscillated approaching the second saddle point. The noise makes it difficult to state that it increased with statistical significance, but if it did it could be due to overfitting. I do not know of any result that relates the separation between saddle points, so the delay could be arbitrary. At some point you reach the bottom, of course.

Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting in the wrong direction. The validation error oscillated approaching the second saddle point. The noise makes it difficult to state that it increased with statistical significance, but if it did it could be due to overfitting. I do not know of any result that relates the separation between saddle points, so the delay could be arbitrary. At some point you reach the bottom, of course.

Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting. The validation error in the SE algorithm oscillated approaching the second saddle point. The noise makes it difficult to state that it increased with statistical significance, but if it did it could be due to overfitting. I do not know of any result that relates the separation between saddle points, so the delay could be arbitrary. At some point you reach the bottom, of course.

Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting in the wrong direction. The validation accuracy increases during plateaus while the training accuracy does not because the model is regularizing (compressing for better generalization) while it seeks a better "regime". I'm just getting to grips with this phenomenon so I'll point you to the source: Opening the Black Box of Deep Neural Networks via Information (presentation)

Because the smaller learning rate allows the optimizer to escape saddle points, which is what happens at each cliff, instead of overshooting in the wrong direction. The validation error oscillated approaching the second saddle point. The noise makes it difficult to state that it increased with statistical significance, but if it did it could be due to overfitting. I do not know of any result that relates the separation between saddle points, so the delay could be arbitrary. At some point you reach the bottom, of course.