Below is a toy example of a CNN that I am trying out. As is observed, the accuracy at the beginning of the first epoch is at 84% and it increases to 96% by the end.
With my understanding of backpropagation, the accuracy is expected to be the same when the next epoch begins - but that does not follow. The accuracy jumps to 98% (from 96%) at the beginning of the second epoch and then marginally increases by the end of it.
What explains the jump?
Epoch 1/10 136/1875 [=>............................] - ETA: 51s - loss: 0.5260 - accuracy: 0.8419 Epoch 1/10 1875/1875 [==============================] - 55s - loss: 0.1315 - accuracy: 0.9606 Epoch 2/10 64/1875 [=>............................] - ETA: 51s - loss: 0.0550 - accuracy: 0.9839 Epoch 2/10 1875/1875 [==============================] - 54s - loss: 0.0447 - accuracy: 0.9864
As another example, I have observed that the accuracy starts at, say, 95.88% at the beginning of an epoch, which is already higher than the end of the previous epoch. And as the epoch training progresses, the accuracy gradually decreases slightly down to, say, 95.52%.
I understand that overfitting and non-shuffled data that is responsible here. But are those the only reasons for this gradual decrease over a single epoch?