Learning rate decay - starting with a higher learning rate for fast convergence and then decreasing the learning rate for better convergence - allows training loss to converge to the same value in less epochs compared to fixing the learning rate to the lower value. However, I noticed better results on my test set when I used a small and constant learning rate and allowed more epochs (training loss reaches the same value, but validation/test loss is better).

Is this a known drawback of SGD with learning rate decay? Is there a way to achieve better generalization without fixing the learning rate to a small value and waiting many epochs?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.