I have implemented a convolutional neural network in Keras, and I use off-line data augmentation in the training set. The way I do this is that I create batches of training data in separate files (because they are not generated in Python), I load one batch/file at a time and run a single epoch of training on it, so during training the exact same data is never seen twice:

for current_batch in range(1, number_of_batches + 1):
    # Load new (X_training, Y_training) pair.

    # Fit for a single epoch.
    model.fit(X_training, Y_training, epochs = 1, batch_size = 32, validation_data = (X_validation, Y_validation))

My understanding is that this is what Keras would do internally if I used it's native (image) data augmentation tools, namely, for each epoch, generate a brand new set of data. I don't use validation split as can be seen above, I rather supply (non-augmented) validation data manually, which is currently the same for each training batch.

My question is this: What is the expected behaviour of (binary cross-entropy) loss and accuracy as a function of the number of batches, in a setting like this? If I performed training for $N >> 1$ epochs on the same training set, there would be a chance of obtaining a very small training loss, whereas in this case, my guess is that I cannot expect that? Indeed, from what I've seen so far, the training loss and accuracy are a bit noisy and how to use stopping criteria etc seems not so easy.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.