I want to solve a task using a ResNet in keras and tensorflow. My Dataset is big, and right now I'm considering my data loading options and trying to determine which one suits the task best.
About the Dataset:
- x: arrays of 200x700 cells in range -1.0...1.0, I don't want to downsample them; they are currently saved as matlab or npz file
- y: the label consists of two floats per x. I have 1.2Million of these (x, y) which are currently saved in 1000 npz files, each with 1GB, totalling to 1TB of data.
Problem: I don't have 1TB RAM in my system, so I can't keep all the data in memory. Thus I need a suitable solution to read my data from disk while training my neural network.
Solutions that I found so far:
- save these files as images and use keras dataset io "load_images_from_directory", downside: I need to save the images on disk which would probably take even more than 1TB. And what about the labels? Plus probably additional preprocessing to from range 0..1 to -1..1
- tfrecords which feels like an overkill, since my dataset is not really a structured one but it's just (array, label)
- hdf files which is also more for structured/hierarchical data.
Things that I also want to take into account:
- Do I save my data as is, or do I need to save shuffled batches? But according to this I should also shuffle the mini-batches in each epoch new. This would mean that the order and filesizes (e.g. one file is one mini-batch) whith which I save my files is not important - the mini-batches should be shuffled anyways.
- Later, I will most likely also need to transfer the whole project to pytorch, so a data storage which is supported by both (kears/tensorflow and pytorch) can save me some time later.
- If I store each (x,y) sample as one small .bin file, this file is smaller than the block size of my disk, thus using more disk size than necessary.
So the question is: What are the pros and cons specific to my dataset/task, thus which dataloading should I use? Are there more options that I haven't discovered yet?