I'm having this burning question in my head, and I couldn't find the answer anywhere. During training, at least in Keras, the training loss is computed on the current batch, so the weights can be updated. So, at least at the first epoch, every batch loss is computed before the model actually learns from that particular epoch. Given this, shouldn't the validation loss for the first epoch be somewhat near the training loss for the first epoch, since both are computed on examples not seen by the gradient descent algorithm? For every model I've built so far, the training loss is always lower than the validtion loss. I expected the training loss to be somewhat near the validation loss on the first run through the dataset, and then, as the model learns from the training dataset (but not from the validation dataset), those errors would start to create a gap between them. Am I missing something trivial here?
$\begingroup$
$\endgroup$
asked Apr 23, 2020 at 9:48
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
Is your batch size the length of the full dataset? If you have $N$ samples, and feed in mini-batches of size $k<N$, then batch losses are computed and the model's weights are updated with every $k$ samples. By the end of the first epoch, there may already have been significantly many updates (learning) from the training set. I believe Keras aggregates these batch losses to compute the epoch loss.
If $k=N$ however, it is possible that the distribution of your training and validation sets are different.
-
$\begingroup$ Yes, I'm using the mini-batch approach. But before the first epoch is finished, each loss is computed in a new mini-batch, not seen before by the algorithm. So, for the losses computed on the first epoch, the model is still learning from new examples, thus my question. $\endgroup$ Commented Apr 23, 2020 at 10:43
-
$\begingroup$ The network learns with every mini-batch, not with every epoch. An epoch simply means every sample in the training data has had a chance to update the model's weights. So let's say you are only halfway through training the first epoch, by then the model will have already had its weights updated N/(2*k) times. The model is already fitted to this first half of your training set, and validation loss will likely be higher. Now if the second half of the training set is completely different, then it's possible that you see some different results by the end of the first epoch. $\endgroup$– AdamCommented Apr 23, 2020 at 10:59
-
1$\begingroup$ That's exactly what I mean, the second half of the training set IS different from the first half, since I'm considering one epoch to be a single pass through the whole dataset. This way, every computed loss, before the first epoch is finished, is produced on a batch not seen by the model, thus it should be at leats close to a loss computed on the cross-validation dataset. $\endgroup$ Commented Apr 23, 2020 at 11:29
-
$\begingroup$ That's interesting you have observed very different losses. It seems to imply that during the training of the first epoch, the model's partial fit on seen data seems to generalize better on the unseen training data versus the unseen validation data. What you've observed isn't really the case all the time. How are you selecting the validation data - randomly sampling? It's possible that each sample/batch is not i.i.d., and that your sampling method is affected by some external factor (such as a persistent time dimension). $\endgroup$– AdamCommented Apr 23, 2020 at 12:30
-
$\begingroup$ Yes, I selected the validation data randomly, so far with a seed. I thought about the possibility that maybe my validation set is 'harder' than my training set, but I selected a different seed, and got the same results. Maybe I should try different seeds and see if this behavior continues. Thanks! $\endgroup$ Commented Apr 23, 2020 at 12:35