As you are working on image classification and would also like to implement some data augmentation, you can combine the two AND load the batches directly from a folder using the mighty 'ImageDataGenerator` class.
Have a look at the execellent documentation!
I won't copy and paste the example from that link, but I can outline the steps that you go through:
create the generator class: data_gen = ImageDataGenerator()
If you want it to perform on-the-fly augmentation for you, that can be specified when creating the class: data_gen = ImageDataGenerator(samplewise_center=True, ...)
If you use augmentation processes that require some statistics about the dataset, e.g. feature_wise normalisation (not sample-wise), you must prepare the generator by showing it some of your data: data_gen.fit(training_data)
. This fit
method simply precomputes things like the mean and standard deviation, which is later used for normalisation.
the generator goes into the model's fit_generator
method, and we call the flow_from_directory
method of the generator:
model.fit_generator(training_data=data_gen.flow_from_directory('/path/to/folder/'), ...)
You can also create a separate generator using ImageDataGenerator for your validation data, where you should then not apply the augmentation, so that validation tests are done on real data, giving you and accurate picture of the model's performance.
In any case, these generators will theoretically run forever, generating batches from your folder. Therefore I recommend using a callback function from Keras to stop when a certain criteria is met. See the documentation for the EarlyStopping class. You could also do this manually, but Keras makes it very simple!
If you want fine-grained control, you could do all of the above manually, loading enough samples from disk for a single batch, performing some augmentation and then running the model.train_on_batch()
method. If you want to get into the details, you might be best first learning the Keras way, then progress to your own detailed models combing Tensorflow as required. The two can be used toegether very nicely!