I am working on a convolutional neural network for image classification. The training dataset is too large to be loaded on my computer memory (4gb), on top of that I also need to try some augmentation to balance the classes.

I am using keras. I have looked into many examples but I have not found a solution to my problem. I am thinking about replacing the parameters of the model.fit call by passing a function or generator doing the 'batching'. This function I will design to import from disk and apply augmentation.

I have no idea how to implement that. Any suggestions?

  • $\begingroup$ Can you show me how to save those train features and their corresponding train lables in disk, so that the next time I would not need to compute the same? $\endgroup$ Commented Jul 1, 2018 at 6:19

1 Answer 1


As you are working on image classification and would also like to implement some data augmentation, you can combine the two AND load the batches directly from a folder using the mighty 'ImageDataGenerator` class.

Have a look at the execellent documentation!

I won't copy and paste the example from that link, but I can outline the steps that you go through:

  1. create the generator class: data_gen = ImageDataGenerator()

  2. If you want it to perform on-the-fly augmentation for you, that can be specified when creating the class: data_gen = ImageDataGenerator(samplewise_center=True, ...)

  3. If you use augmentation processes that require some statistics about the dataset, e.g. feature_wise normalisation (not sample-wise), you must prepare the generator by showing it some of your data: data_gen.fit(training_data). This fit method simply precomputes things like the mean and standard deviation, which is later used for normalisation.

  4. the generator goes into the model's fit_generator method, and we call the flow_from_directory method of the generator:

    model.fit_generator(training_data=data_gen.flow_from_directory('/path/to/folder/'), ...)

You can also create a separate generator using ImageDataGenerator for your validation data, where you should then not apply the augmentation, so that validation tests are done on real data, giving you and accurate picture of the model's performance.

In any case, these generators will theoretically run forever, generating batches from your folder. Therefore I recommend using a callback function from Keras to stop when a certain criteria is met. See the documentation for the EarlyStopping class. You could also do this manually, but Keras makes it very simple!

If you want fine-grained control, you could do all of the above manually, loading enough samples from disk for a single batch, performing some augmentation and then running the model.train_on_batch() method. If you want to get into the details, you might be best first learning the Keras way, then progress to your own detailed models combing Tensorflow as required. The two can be used toegether very nicely!

  • $\begingroup$ I did not understand 3rd step. Can you tell me,in 3rd step training_data should whole data set or some data from Data-set ? $\endgroup$
    – Junaid
    Commented Feb 5, 2018 at 13:00
  • $\begingroup$ @Junaid - It can be a small part of the data. It needs to be enough, however, to calculate a reasonable value of e.g. the mean, which is used then for the entire dataset during training. I have added a little more information to point 3. Here is a link to an example of the flow_from_directory method $\endgroup$
    – n1k31t4
    Commented Feb 6, 2018 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.