I know that overfitting occurs when the accuracy on the training set improves but the accuracy on the validation set decrease. So, we must stop the training. I would like to know if this is a rule that more epochs always decrease the accuracy on the training set. In other words, is the accuracy function over epochs a non-ascending function? If not, then why?


1 Answer 1


Yes, accuracy measured on the training set over epochs is not monotonically decreasing.

Possible reasons:

  1. Software bug
  2. Nan due to division by zero / log (0) / overflow
  3. Too big learning rate

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.