2
$\begingroup$

The rationale behind the keras function ReduceLROnPlateau() is that models benefit from reducing learning rate once learning stagnates.

Is it a good practice to always apply ReduceLROnPlateau()? What are some situations, if any, to not apply ReduceLROnPlateau()?

$\endgroup$
2
$\begingroup$

I believe it is, in theory, always a good idea to use this method. I say in theory because the theory of gradient descent points to the fact that a minimum can only be reached when the learning rate approaches 0. Otherwise, with a permanent large learning rate, the model's performance (i.e. the loss metric) will bounce around the minimum – always overshooting it.

That being said, it is important to think practically and be aware of the dangers of using this method. There can be phases of training where you don't improve your metric by much, and so the learning rate is reduced. But this can happen prematurely - depending on the plateau-parameters you selected - and so actually impede/prevent the model from ever getting far enough down the loss curve! If you set a minimum learning rate, you may always get down to a minimum (local or otherwise), but it could take a lot longer if your learning rate was reduced prematurely.

In the end, trying out as many combinations as possible is likely your best best.

Alternatively, you could try combing this with simulated annealing, whereby the learning rate is boosted upwards again, according to a predefined rule, such as after a number of epochs or a %-reduction in the learning rate itself.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.