1
$\begingroup$

We know that if training and test loss are different from each other, our model is over-fitting. However, if both get high after some epochs, how can we justify it?

One way to solve it is to reduce the learning rate to 0.0001.

But, actually I would like to know the theoretical reason behind this sudden increment.

accuracy and loss

$\endgroup$
0
0
$\begingroup$

This might be happening because of high learning rate.

The loss function is convex and the model needs to reach the minima. Learning rate defines the magnitude of step you take towards minima. If you take high learning rate, you might overshoot the minima, hence sudden increase in loss value.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.