When using the early stopping callback in Keras, training stops when some metric (usually validation loss) is not increasing. Is there a way to use another metric (like precision, recall, or f-measure) instead of validation loss?

All the examples I have seen so far are similar to this one:

callbacks.EarlyStopping(monitor='val_loss', patience=5, verbose=0, mode='auto')

You can use any metric function that you specified when compiling the model.

Let's say you have the following metric function:

def my_metric(y_true, y_pred):
     return some_metric_computation(y_true, y_pred)

The only requirement to this function is that it takes accepts the true y and the predicted y.

When you compile the model, you specify this metric, similarly to how you specify build in metrics like 'accuracy':

model.compile(metrics=['accuracy', my_metric], ...)

Notice that we are using the function name my_metric without ' ' (in contrast to the build in 'accuracy').

Then, if you define your EarlyStopping, just use the name of the function (this time with ' '):

EarlyStopping(monitor='my_metric', mode='min')

Make sure to specify the mode (min if lower is better, max if higher is better).

You can use it just like any build-in metric. This probably also works with other Callbacks like ModelCheckpoint (but I have not tested that). Internally, Keras just adds the new metric to the list of metrics available for this model using the function name.

If you specify data for validation in your model.fit(...), then you can also use it for EarlyStopping by using 'val_my_metric'.


Of course, just create your own!

class EarlyStopByF1(keras.callbacks.Callback):
    def __init__(self, value = 0, verbose = 0):
        super(keras.callbacks.Callback, self).__init__()
        self.value = value
        self.verbose = verbose

    def on_epoch_end(self, epoch, logs={}):
         predict = np.asarray(self.model.predict(self.validation_data[0]))
         target = self.validation_data[1]
         score = f1_score(target, prediction)
         if score > self.value:
            if self.verbose >0:
                print("Epoch %05d: early stopping Threshold" % epoch)
            self.model.stop_training = True

callbacks = [EarlyStopByF1(value = .90, verbose =1)]
model.fit(X, y, batch_size = 32, nb_epoch=nb_epoch, verbose = 1, 
validation_data(X_val,y_val), callbacks=callbacks)

I haven't tested this out but that should be the general flavor for how you go about it. If it doesn't work let me know and i'll retry over the weekend. I'm also assuming you have your own f1 score implemented already. If not just import for sklearn.

  • 1
    $\begingroup$ +1 Still works as of 2/11/2020 using latest Keras and Python 3.7 $\endgroup$ – Austin Feb 11 '20 at 23:32
  • $\begingroup$ @Austin could you share the code? I tried to implement it but it throws some errors for me. $\endgroup$ – Luis Miguel Apr 8 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.