In theory and what people are doing (e.g. Paper) decreasing the learning rate should help the optimizer to go "deeper into the valley" and thus decrease the loss and increase the metric.

Thus, my plan was to train a neural network with a learning rate of 1 until the loss and my metric stay approx. the same for some epochs, then with 0.1, then 0.01 and so on.

However, what I'm observing is, that the loss of the model stagnates after 1000 epochs with learning rate 1 and then decreasing the learning rate multiple times doesn't improve anything.

Do I have to decrease the lr earlier, e.g. after 400 epochs when the neural network is still learning and the loss is going down? Or do you have any ideas why this happens?

Thank you!

  • $\begingroup$ If you are to do this, then it would be better to do it smoothly and continuously, rather than jumping from one alpha to the next abruptly, in big leaps, after the model has already seemingly converged somewhere. $\endgroup$ Commented Nov 5, 2021 at 10:16
  • 1
    $\begingroup$ Why is it better? Is there any "proof"? Or just common use? $\endgroup$
    – wuiwuiwui
    Commented Nov 6, 2021 at 11:49

1 Answer 1


You might want to have a look at several ways of scheduling your learning rate decay, instead of manually trying to optimize it. Check this source from Tensorflow documentation. If you for instance use an exponential learning rate decay schedule, the definition for the learning rate decay would be:

$$learning rate_0 * (decay rate)^\frac{step}{decay steps}$$

but you can try out other schedules too, among which you can find the piecewise constant decay as a simpler one to begin with.

  • $\begingroup$ Thank you, I'll check it :) $\endgroup$
    – wuiwuiwui
    Commented Nov 6, 2021 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.