Is shuffling data really necessary for training? [duplicate]

Question

I don't mean if we had a dataset where if sequentially sampled, the labels would be [1111122223333]. In this case, the network learns to predict everything as 1, then 2, and so on and it's impossible to learn.

I mean: Assume you have Imagenet 2012 dataset. You shuffle it once. So now the labels and the images are shuffled. Since the dataset is huge, can the network really remember the previous epoch's predictions and overfit?

OR, I shuffle data 5 times and use each ordering in epochs 1,2,..5, and then at epoch 6 I use the Shuffled data#1 again.

Everybody talks about the importance of shuffling but I never read anything that addresses these problems.

BTW, this question was prompted by me using a database where accessing data sequentially is a lot faster than random access. If I knew that even a pseudo-shuffling helps, it would save me 6-7 hours per training epoch.

Answer 1

As well as extreme confounding examples like the one you mention in your question, the thing with shuffling is reproducibility. If you accidentally discover a certain order that gives better than average results, if you publish those results they could be misleading.

In your case I’m not sure what the answer if as I’m not sure what algorithm(s) you are using, what sort of database, etc. Maybe you could add these details to your question. In principle, most databases should allow you to retrieve records really quickly as long as your search is supported by an index, so you could probably generate a random shuffle of your index outside of your database to determine the order in which you retrieve training examples.

If you really do need to only use a small number of preset shuffles, 2 is probably fine. Use statistical tests like this one to show that sufficiently large consecutive queries using each ordering behave like samples from the same population.