# Shouldn't training and validation loss be approximate before the first epoch is finished?

I'm having this burning question in my head, and I couldn't find the answer anywhere. During training, at least in Keras, the training loss is computed on the current batch, so the weights can be updated. So, at least at the first epoch, every batch loss is computed before the model actually learns from that particular epoch. Given this, shouldn't the validation loss for the first epoch be somewhat near the training loss for the first epoch, since both are computed on examples not seen by the gradient descent algorithm? For every model I've built so far, the training loss is always lower than the validtion loss. I expected the training loss to be somewhat near the validation loss on the first run through the dataset, and then, as the model learns from the training dataset (but not from the validation dataset), those errors would start to create a gap between them. Am I missing something trivial here?

Is your batch size the length of the full dataset? If you have $$N$$ samples, and feed in mini-batches of size $$k, then batch losses are computed and the model's weights are updated with every $$k$$ samples. By the end of the first epoch, there may already have been significantly many updates (learning) from the training set. I believe Keras aggregates these batch losses to compute the epoch loss.

If $$k=N$$ however, it is possible that the distribution of your training and validation sets are different.

• Yes, I'm using the mini-batch approach. But before the first epoch is finished, each loss is computed in a new mini-batch, not seen before by the algorithm. So, for the losses computed on the first epoch, the model is still learning from new examples, thus my question. Apr 23, 2020 at 10:43
• The network learns with every mini-batch, not with every epoch. An epoch simply means every sample in the training data has had a chance to update the model's weights. So let's say you are only halfway through training the first epoch, by then the model will have already had its weights updated N/(2*k) times. The model is already fitted to this first half of your training set, and validation loss will likely be higher. Now if the second half of the training set is completely different, then it's possible that you see some different results by the end of the first epoch.