I have a highly imbalanced dataset with less than 0.5% of the minor class. Using Keras, I'm training DNN on the training set and evaluate performance on validation set. Loss function is
I'm setting my early-stopping on f1 score, instead of validation loss. What I observe during training is f1 score fluctuate wildly up and down while validation loss is decreasing. I actually end up with a very low f1 score with early-stopping, although f1 score was many epochs back...
I'm confused about it. Am I supposed to do early-stopping on the performance metrics? Should we alway use validation loss for early-stopping criterion? thanks.