How to best read large dataset from disk

Question

I want to solve a task using a ResNet in keras and tensorflow. My Dataset is big, and right now I'm considering my data loading options and trying to determine which one suits the task best.

About the Dataset:

• x: arrays of 200x700 cells in range -1.0...1.0, I don't want to downsample them; they are currently saved as matlab or npz file
• y: the label consists of two floats per x. I have 1.2Million of these (x, y) which are currently saved in 1000 npz files, each with 1GB, totalling to 1TB of data.

Problem: I don't have 1TB RAM in my system, so I can't keep all the data in memory. Thus I need a suitable solution to read my data from disk while training my neural network.

Solutions that I found so far:

• save these files as images and use keras dataset io "load_images_from_directory", downside: I need to save the images on disk which would probably take even more than 1TB. And what about the labels? Plus probably additional preprocessing to from range 0..1 to -1..1
• tfrecords which feels like an overkill, since my dataset is not really a structured one but it's just (array, label)
• hdf files which is also more for structured/hierarchical data.

Things that I also want to take into account:

• Do I save my data as is, or do I need to save shuffled batches? But according to this I should also shuffle the mini-batches in each epoch new. This would mean that the order and filesizes (e.g. one file is one mini-batch) whith which I save my files is not important - the mini-batches should be shuffled anyways.
• Later, I will most likely also need to transfer the whole project to pytorch, so a data storage which is supported by both (kears/tensorflow and pytorch) can save me some time later.
• If I store each (x,y) sample as one small .bin file, this file is smaller than the block size of my disk, thus using more disk size than necessary.

So the question is: What are the pros and cons specific to my dataset/task, thus which dataloading should I use? Are there more options that I haven't discovered yet?

Answer 1

A common way is to create a class inheriting tf.keras.utils.Sequence. This class implements a function __getitem__ which is called when you use model.fit() method. In this method, you can simply load one batch at a time, so no need to load the whole dataset. See the documentation. You can also directly use the .npz files when calling __getitem__.

Shuffle data

Do I save my data as is, or do I need to save shuffled batches?

You would implement the method on_epoch_end() that does the shuffling of a list of indices. Then when you load a .npz file, use something like data[indices[i]] which load the index i in the shuffled list of indices. on_epoch_end is called at the end of each epoch by .fit() method, and you can also use it in __init__ of your Sequence class to initialize the shuffle.

From Keras to Pytorch

Pytorch has similar module called torch.utils.data.Dataset. The conversion is straightforward. See this tutorial.