Can accuracy become worse on the training set with more epochs?

Question

I know that overfitting occurs when the accuracy on the training set improves but the accuracy on the validation set decrease. So, we must stop the training. I would like to know if this is a rule that more epochs always decrease the accuracy on the training set. In other words, is the accuracy function over epochs a non-ascending function? If not, then why?

Answer 1

Yes, accuracy measured on the training set over epochs is not monotonically decreasing.

Possible reasons:

1. Software bug
2. Nan due to division by zero / log (0) / overflow
3. Too big learning rate