1
$\begingroup$

So I want to stop the cnn when a custom (not implemented in keras) logged metric is not improving with a patience of 5 (I chose macro f1 score) and here's what I did:

Created a callback to log the macro f1 score on epoch end and an early stopping:

early_stopping = EarlyStopping(monitor='val_macro_f1', patience=5, restore_best_weights=True)
macro_f1_callback = MacroF1Callback(x_valid_combined_tfidf, y_valid)

And here is the (simplified) output of the fitting:

Epoch 1/20
Validation Macro F1 Score: 0.3983

Epoch 2/20
Validation Macro F1 Score: 0.3369

Epoch 3/20
Validation Macro F1 Score: 0.4057

Epoch 4/20
Validation Macro F1 Score: 0.3947

Epoch 5/20
Validation Macro F1 Score: 0.3761

Epoch 6/20
Validation Macro F1 Score: 0.3918

Epoch 7/20
Validation Macro F1 Score: 0.4147  <keras.src.callbacks.History at 0x4cae76210>

And after predicting again on the validation data, it seems that the early stopping chose the better weights to be from...epoch 2...

F1 score: 0.33687923314086654

All this doesn't make sense, because ok, epoch 2 started decreasing the metric since its lower then epoch 1, and it ends in epoch 7, but epoch 7 has a metric even better the epoch 1

Can anyone help me with this? Maybe I'm doing something wrong. Also please tell me if you want to paste some more code here.

Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

In my view, all happened as it was supposed to. EarlyStopping in the 7th epoch concluded: "The lowest metric was seen 5 epochs ago and the model stopped improving, so let's stop the training" - exactly how you defined it.

Your problem is that you want to decrease and not increase your custom metric. However, if your goal is to increase some function, it's not a loss function. If I understood it correctly, you can try putting a minus before the F1 score to let the model decrease its negative equivalent or give a try to other similar loss functions like categorical cross-entropy.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks! I realized what I was doing wrong, I figured the mode='auto' from early stopping would be better, but now that I specified which way the metric should go, it works fine. Tysm for pointing me in the right direction :) $\endgroup$
    – giza2001s
    Dec 19, 2023 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.