Currently, I use image files and transform them into a *.npy file(saved as a numpy array) as training data. At present this training data set is nearly 3GB. Now I have more image files, so the training data set will become even larger, maybe up to 40 GB. And I am afraid that the *.npy files can not save this big of a data file.

Is there another possible way to store such a large file?

Since I use Keras to build a neural network model, is it acceptable to split the training data into small parts so as to train the model without having to use all the training data?

Does this sound like a reasonable approach?


2 Answers 2


I do something similar with keras and GPU training, where i also have only a small memory amount available. The idea would be split the numpy files into smaller ones, let's say 64 samples per file and then load each file and call train_on_batch on those images. You can use keras' train_on_batch function to achieve this:


train_on_batch(self, x, y, class_weight=None, sample_weight=None)

It's designed to perform a single gradient update over one batch of samples.

Custom Generator

Another idea is to use generators which provide you with data given a directory. They can also be used for data augmentation, i.e. randomly generating new training data from your data. Here is an example from the keras documentation:

train_datagen = ImageDataGenerator(

# just rescale test data
test_datagen = ImageDataGenerator(rescale=1./255)

# this generator loads data from the given directory and 32 images 
# chunks called batches. you can set this as you like
train_generator = train_datagen.flow_from_directory(
        target_size=(150, 150),

# same es the train_generator    
validation_generator = test_datagen.flow_from_directory(
        target_size=(150, 150),

# loads sequentially images and feeds them to the model. 
# the batch size is set in the constructor 

Concerning the size of the data - have you tried compressing the files using the Python tarfile library? You could compress it in chunks and after each successful compression operation, and keep a sql db with the metadata in it. That way when you do your sample selection you can perform the selection on the db and pull only the files you need.

I would go for it and split it up. As far as a selection method I would kick it old school and do a random selection with replacement.

  • 1
    $\begingroup$ "too large a training sample can result in overfitting" - it's the other way around :) $\endgroup$
    – stmax
    Commented Mar 9, 2017 at 11:15
  • $\begingroup$ Thanks for the catch. My brain must have been going one way while my hands went the other. I guess it really just depend on how many features he is using. $\endgroup$
    – Dan Temkin
    Commented Mar 9, 2017 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.