0
$\begingroup$

Learning rate decay - starting with a higher learning rate for fast convergence and then decreasing the learning rate for better convergence - allows training loss to converge to the same value in less epochs compared to fixing the learning rate to the lower value. However, I noticed better results on my test set when I used a small and constant learning rate and allowed more epochs (training loss reaches the same value, but validation/test loss is better).

Is this a known drawback of SGD with learning rate decay? Is there a way to achieve better generalization without fixing the learning rate to a small value and waiting many epochs?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.