How do you normally organize large data-sets for easy loading when training Neural Networks? I have a largeish data-set which cannot fit into memory, it consists of 200,000 samples, with 10k samples being stored in grouped files. In total this is roughly 50GB of data.
I can separate the grouping to produce 200,000 individual files but I'm not sure if this is the correct course of action as the system will need to do many calls to the file system when training.
How do systems which train massive data-sets work? (For example image-net).