When working with very large models within Deep Learning, training often takes long and requires small batch sizes due to memory restrictions.
Usually, we are left with a model checkpoint after training has commenced. I am wondering whether the exact time at which we take that checkpoint significantly factors in to the statistical properties of a model's outputs.
Within text generation, lets assume that just before we extract the checkpoint, the model learns statistically anomalous batches with longer sentences than the mean.
Would that result in our model generating longer sentences, overrepresenting that recent batch of anomalous texts?
As training batches are often randomly generated from the dataset, such unrepresentative batches may certainly occur, sometimes right before we save the checkpoint.
Has there been any research regarding such, potentially unwanted, recency bias in slower deep learning scenarios?
The only references I could find were intentionally trying to employ such biases, but I have not found any literature on unwanted recency bias.