When training NN, how do data loaders work on large datasets?

Question

How do you normally organize large data-sets for easy loading when training Neural Networks? I have a largeish data-set which cannot fit into memory, it consists of 200,000 samples, with 10k samples being stored in grouped files. In total this is roughly 50GB of data.

I can separate the grouping to produce 200,000 individual files but I'm not sure if this is the correct course of action as the system will need to do many calls to the file system when training.

How do systems which train massive data-sets work? (For example image-net).

Answer 1

For large data sets the limitations of available memory prevents you from loading all the data simultaneously. What is typically done is to provide your data to the network in batches. Batches are simply groupings of your data. The batch size maximum value is limited by available memory. If you use Keras the ImageDataGenerator.flow from directory provides a convenient way to present your data to the network in batches. Documentation is here. There is also a benefit to presenting your data in batches. For example if your data consists of 50,000 samples and it fits into memory you can provide the 50,000 samples to the network and after the data is forward passed do 1 step of back propagation to adjust the weights. A better solution is to partition your data into batches, say 10 batches of 5,000 samples. In this case if the network is enabled to train on batches you can get 10 steps of iteration of the weights versus the single iteration. This enables the network to converge faster at a lower computational cost.