0
$\begingroup$

I just trained a model (SGD) with keras and was wondering why the change of accuracy and loss from epoch to epoch doesn't really decrease that much when I lower the learning rate. So I tested what happens when I set the learning rate to 0 and to my surprise, accuracy and loss still changed from epoch to epoch and I can't find an explanation for that. Does anyone know why this could be happening?

$\endgroup$
1
$\begingroup$

If your learning rate is set lower, training will progress very slowly because you are making very tiny updates to the weights. However, if you set learning rate higher, it can cause undesirable divergent behavior in your loss function. So when you set learning rate lower you need to set higher number of epochs.

The reason for change when you set learning rate to 0 is beacuse of Batchnorm. If you have batchnorm in your model, remove it and try. Look at these link, link

$\endgroup$
1
  • $\begingroup$ Thanks, it must have been the batch normalization. After removing it, it does not seem to change anymore. $\endgroup$
    – Evator
    Feb 18 '21 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.