6
$\begingroup$

I have a highly imbalanced dataset with less than 0.5% of the minor class. Using Keras, I'm training DNN on the training set and evaluate performance on validation set. Loss function is binary_crossentropy.

I'm setting my early-stopping on f1 score, instead of validation loss. What I observe during training is f1 score fluctuate wildly up and down while validation loss is decreasing. I actually end up with a very low f1 score with early-stopping, although f1 score was many epochs back...

I'm confused about it. Am I supposed to do early-stopping on the performance metrics? Should we alway use validation loss for early-stopping criterion? thanks.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

F1 is based on hard classification; if the probability scores are hovering near the threshold, then the classifications may be flopping a lot, leading to unstable F1 scores.

A low F1 score is not too surprising in the presence of such imbalance; the default cutoff of 0.5 will often lead to high recall but low precision.

Improve this answer
$\endgroup$
0
$\begingroup$

One possible problem might be that Precision and Recall are very different.

In general, F1 is only useful if Precision and Recall are similar. If they are very unbalanced, it will screw up your F1 score. So early stopping might be better based on Precision or Recall.

Attention: Keras loves normalization of the data. But if you have a lot of outliers in your dataset, you squeeze them into the normalization where they will dominate.

Improve this answer
$\endgroup$
3
  • $\begingroup$ What do you mean by: a) ... it will screw up your F1 score. and b) Keras loves normalization of the data? $\endgroup$
    – naive
    Feb 23, 2019 at 9:03
  • $\begingroup$ A: Roughly speaking, F1 is the average of Precision and Recall. If Precision and Recall are very different, you can get a high fluctuation in F1. B: Francois Chollet, the creator of Keras, strongly recommends normalizing your data before feeding it to a neural network. If you use SKlearn to preprocess, this is the MinMaxScaler. If you have outliers in your dataset, you squeeze them into 0 - 1 where they will dominate. That's one reason why you don't see on Kaggle winners using Keras/TensorFlow on tabular datasets (nominal + categorical data). $\endgroup$
    – FrancoSwiss
    Feb 23, 2019 at 9:54
  • $\begingroup$ Outliers will cause problems regardless of scaling method, but maybe StandardScaler will be better? $\endgroup$
    – Ben Reiniger - on strike ♦
    Mar 19, 2020 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.