Why do we use the F1 score instead of mutual information?

We often use the classification threshold that maximizes the F1 score, if we don't have a prior cost function of false positives and false negatives.

This balances the desire for precision and recall. If either one is 0 the F1 score is 0; and if we have a perfect classification the F1 score is 1.

On the other hand I'm hard pressed to find a scientific justification to maximize F1 in general, or a business problem where F1 is the thing we need to maximize.

F1 is not symmetric. If we have an 60/40 binary distribution and choose the 40% class as the positive class, and we classify everything as positive, we get 100% recall and 40% precision for F1 score of 0.4. (F1=0 if we classify everything negative). If we choose the 60% class as positive, and classify everything positive, we get F1 score of 0.6.

Why not use mutual information, which minimizes the surprise in the prediction vs. actual?

When we estimate probabilities by minimizing log loss, we are also minimizing K-L divergence or entropy or surprise in the information theory sense. If a 50/50 probability prediction contains 1 bit of entropy or surprise, minimizing the log loss minimizes the number of bits of entropy or surprise in our predictions vs. actual.

And in information theory, if we don't have probabilities and want to measure the information transmitted by a noisy signal, we use mutual information.

And mutual information is symmetric, and 0 for a signal of all 1s or 0s. Predicting all 1s or 0s in some sense gives no information about the ground truth of the response variable.

In a way, a prediction is like a noisy channel from the present to the future, and an information theory concept like mutual information seems like a well-grounded criterion to choose the classification threshold, and F1 seems arbitrary.

Is there any reason why F1 is preferred over mutual information?

• Honestly, I see virtually no reason why we would want to maximize any improper scoring rule like F1, accuracy, Kappa, etc. when they often lead to non optimal models for reasons that you discussed. Working with probabilities and then optimizing a cost function related to the problem seems way more reasonable, and log loss/Brier score are optimized only when we get probabilities that best match the population. I suppose interpretation might be one, and class imbalance issues? But realistically, we can always find the log loss for any naive classifier that just... Aug 30, 2019 at 17:24
• ...predicts the majority class with probability equal to the MLE as given in the training set and compare it to the log loss of our model to see if there is any benefit. Aug 30, 2019 at 17:27
• @aranglol I agree ... I struggle to see any real-world situation where F1 is the thing to maximize. Mutual information has a theoretical foundation. I can also justify minimizing a cost function related to the relative frequency. If the number of positives / number of samples is p, minimize p * false_positives + (1-p) * false negatives. If you assume slope of ROC curve is continuously decreasing, that would be the point on the ROC curve farthest from the 45º line where slope ~= 1, you trade additional p false positives for additional (1-p) false negatives. Sep 5, 2019 at 20:25