How to train data by batch from disk?

Question

I am working on a convolutional neural network for image classification. The training dataset is too large to be loaded on my computer memory (4gb), on top of that I also need to try some augmentation to balance the classes.

I am using keras. I have looked into many examples but I have not found a solution to my problem. I am thinking about replacing the parameters of the model.fit call by passing a function or generator doing the 'batching'. This function I will design to import from disk and apply augmentation.

I have no idea how to implement that. Any suggestions?

Answer 1

As you are working on image classification and would also like to implement some data augmentation, you can combine the two AND load the batches directly from a folder using the mighty 'ImageDataGenerator` class.

Have a look at the execellent documentation!

I won't copy and paste the example from that link, but I can outline the steps that you go through:

1. create the generator class: data_gen = ImageDataGenerator()

2. If you want it to perform on-the-fly augmentation for you, that can be specified when creating the class: data_gen = ImageDataGenerator(samplewise_center=True, ...)

3. If you use augmentation processes that require some statistics about the dataset, e.g. feature_wise normalisation (not sample-wise), you must prepare the generator by showing it some of your data: data_gen.fit(training_data). This fit method simply precomputes things like the mean and standard deviation, which is later used for normalisation.

4. the generator goes into the model's fit_generator method, and we call the flow_from_directory method of the generator:

   model.fit_generator(training_data=data_gen.flow_from_directory('/path/to/folder/'), ...)

You can also create a separate generator using ImageDataGenerator for your validation data, where you should then not apply the augmentation, so that validation tests are done on real data, giving you and accurate picture of the model's performance.

In any case, these generators will theoretically run forever, generating batches from your folder. Therefore I recommend using a callback function from Keras to stop when a certain criteria is met. See the documentation for the EarlyStopping class. You could also do this manually, but Keras makes it very simple!

If you want fine-grained control, you could do all of the above manually, loading enough samples from disk for a single batch, performing some augmentation and then running the model.train_on_batch() method. If you want to get into the details, you might be best first learning the Keras way, then progress to your own detailed models combing Tensorflow as required. The two can be used toegether very nicely!