Is there away to change the metric used by the Early Stopping callback in Keras?

Question

When using the early stopping callback in Keras, training stops when some metric (usually validation loss) is not increasing. Is there a way to use another metric (like precision, recall, or f-measure) instead of validation loss?

All the examples I have seen so far are similar to this one:

callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='auto')

Answer 1

You can use any metric function that you specified when compiling the model.

Let's say you have the following metric function:

def my_metric(y_true, y_pred):
     return some_metric_computation(y_true, y_pred)

The only requirement to this function is that it takes accepts the true y and the predicted y.

When you compile the model, you specify this metric, similarly to how you specify build in metrics like 'accuracy':

model.compile(metrics=['accuracy', my_metric], ...)

Notice that we are using the function name my_metric without ' ' (in contrast to the build in 'accuracy').

Then, if you define your EarlyStopping, just use the name of the function (this time with ' '):

EarlyStopping(monitor='my_metric', mode='min')

Make sure to specify the mode (min if lower is better, max if higher is better).

You can use it just like any build-in metric. This probably also works with other Callbacks like ModelCheckpoint (but I have not tested that). Internally, Keras just adds the new metric to the list of metrics available for this model using the function name.

If you specify data for validation in your model.fit(...), then you can also use it for EarlyStopping by using 'val_my_metric'.

Answer 2

Of course, just create your own!

class EarlyStopByF1(keras.callbacks.Callback):
    def __init__(self, value = 0, verbose = 0):
        super(keras.callbacks.Callback, self).__init__()
        self.value = value
        self.verbose = verbose


    def on_epoch_end(self, epoch, logs={}):
         predict = np.asarray(self.model.predict(self.validation_data[0]))
         target = self.validation_data[1]
         score = f1_score(target, prediction)
         if score > self.value:
            if self.verbose >0:
                print("Epoch %05d: early stopping Threshold" % epoch)
            self.model.stop_training = True


callbacks = [EarlyStopByF1(value = .90, verbose =1)]
model.fit(X, y, batch_size = 32, nb_epoch=nb_epoch, verbose = 1, 
validation_data(X_val,y_val), callbacks=callbacks)

I haven't tested this out but that should be the general flavor for how you go about it. If it doesn't work let me know and i'll retry over the weekend. I'm also assuming you have your own f1 score implemented already. If not just import for sklearn.