How to deal with large training data?

Question

Currently, I use image files and transform them into a *.npy file(saved as a numpy array) as training data. At present this training data set is nearly 3GB. Now I have more image files, so the training data set will become even larger, maybe up to 40 GB. And I am afraid that the *.npy files can not save this big of a data file.

Is there another possible way to store such a large file?

Since I use Keras to build a neural network model, is it acceptable to split the training data into small parts so as to train the model without having to use all the training data?

Does this sound like a reasonable approach?

Answer 1

I do something similar with keras and GPU training, where i also have only a small memory amount available. The idea would be split the numpy files into smaller ones, let's say 64 samples per file and then load each file and call train_on_batch on those images. You can use keras' train_on_batch function to achieve this:

train_on_batch

train_on_batch(self, x, y, class_weight=None, sample_weight=None)

It's designed to perform a single gradient update over one batch of samples.

Custom Generator

Another idea is to use generators which provide you with data given a directory. They can also be used for data augmentation, i.e. randomly generating new training data from your data. Here is an example from the keras documentation:

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

# just rescale test data
test_datagen = ImageDataGenerator(rescale=1./255)

# this generator loads data from the given directory and 32 images 
# chunks called batches. you can set this as you like
train_generator = train_datagen.flow_from_directory(
        'data/train',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

# same es the train_generator    
validation_generator = test_datagen.flow_from_directory(
        'data/validation',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

# loads sequentially images and feeds them to the model. 
# the batch size is set in the constructor 
model.fit_generator(
        train_generator,
        samples_per_epoch=2000,
        nb_epoch=50,
        validation_data=validation_generator,
        nb_val_samples=800)

Answer 2

Concerning the size of the data - have you tried compressing the files using the Python tarfile library? You could compress it in chunks and after each successful compression operation, and keep a sql db with the metadata in it. That way when you do your sample selection you can perform the selection on the db and pull only the files you need.

I would go for it and split it up. As far as a selection method I would kick it old school and do a random selection with replacement.