How to checkpoint by minibatch in Keras

Question

I know that I can use ModelCheckpoint in Keras for checkpointing a model every epoch (or every few epochs, depending on what I want).

I am getting my data for each minibatch from a fit_generator, and it takes a very long time to evaluate each minibatch. I'd like to be able to checkpoint by minibatch instead of by epoch. How can I do this in Keras?

Answer 1

You have to write a custom callback for this. Steps are :

1. Subclass ModelCheckpoint (https://github.com/keras-team/keras/blob/master/keras/callbacks.py) or create new one if you do not need filename pattern etc.

2. Add method that would be called at the end of each batch

class BatchModelCheckpoint(keras.callbacks.Callback):
     def on_batch_end(self, batch, logs=None):
        self.model.save(filepath, overwrite=True)