6
$\begingroup$

We often use the classification threshold that maximizes the F1 score, if we don't have a prior cost function of false positives and false negatives.

This balances the desire for precision and recall. If either one is 0 the F1 score is 0; and if we have a perfect classification the F1 score is 1.

On the other hand I'm hard pressed to find a scientific justification to maximize F1 in general, or a business problem where F1 is the thing we need to maximize.

F1 is not symmetric. If we have an 60/40 binary distribution and choose the 40% class as the positive class, and we classify everything as positive, we get 100% recall and 40% precision for F1 score of 0.4. (F1=0 if we classify everything negative). If we choose the 60% class as positive, and classify everything positive, we get F1 score of 0.6.

Why not use mutual information, which minimizes the surprise in the prediction vs. actual?

When we estimate probabilities by minimizing log loss, we are also minimizing K-L divergence or entropy or surprise in the information theory sense. If a 50/50 probability prediction contains 1 bit of entropy or surprise, minimizing the log loss minimizes the number of bits of entropy or surprise in our predictions vs. actual.

And in information theory, if we don't have probabilities and want to measure the information transmitted by a noisy signal, we use mutual information.

And mutual information is symmetric, and 0 for a signal of all 1s or 0s. Predicting all 1s or 0s in some sense gives no information about the ground truth of the response variable.

In a way, a prediction is like a noisy channel from the present to the future, and an information theory concept like mutual information seems like a well-grounded criterion to choose the classification threshold, and F1 seems arbitrary.

Is there any reason why F1 is preferred over mutual information?

$\endgroup$
  • $\begingroup$ Honestly, I see virtually no reason why we would want to maximize any improper scoring rule like F1, accuracy, Kappa, etc. when they often lead to non optimal models for reasons that you discussed. Working with probabilities and then optimizing a cost function related to the problem seems way more reasonable, and log loss/Brier score are optimized only when we get probabilities that best match the population. I suppose interpretation might be one, and class imbalance issues? But realistically, we can always find the log loss for any naive classifier that just... $\endgroup$ – aranglol Aug 30 at 17:24
  • $\begingroup$ ...predicts the majority class with probability equal to the MLE as given in the training set and compare it to the log loss of our model to see if there is any benefit. $\endgroup$ – aranglol Aug 30 at 17:27
  • $\begingroup$ @aranglol I agree ... I struggle to see any real-world situation where F1 is the thing to maximize. Mutual information has a theoretical foundation. I can also justify minimizing a cost function related to the relative frequency. If the number of positives / number of samples is p, minimize p * false_positives + (1-p) * false negatives. If you assume slope of ROC curve is continuously decreasing, that would be the point on the ROC curve farthest from the 45º line where slope ~= 1, you trade additional p false positives for additional (1-p) false negatives. $\endgroup$ – Rocky McNuts Sep 5 at 20:25
3
$\begingroup$

The reason why one uses any particular evaluation measure should be based on the semantics of the task. Corollary: there's no unique evaluation measure which is perfect for every task. Obviously there are also technical constraints to take into account, but imho considering only the technical aspect of performance optimization is the most common mistake made in ML applications.

For instance OP mentions non-symetry of the F1-score: in some tasks it makes complete sense to use a non-symmetric measure, for example any task where the goal is to extract specific elements, e.g. named entity recognition. Moreover the question of symmetry makes sense only for binary classification, and many tasks involve more than 2 possible classes. In such cases it's common to use micro or macro F-score, but again it depends what one wants to measure. It's also important to keep in mind that a single metric is always a simplification of the performance, often there are many other relevant aspects. For instance, mistakes for a particular class might be more costly than for another, like in the case of medical prediction systems where a false negative means that a patient doesn't get a potentially life-saving treatment (note that generic F-score can be weighted to account for that).

The interpretability of the performance measure is also very important. In many applications one needs to estimate some kind of average error rate that a non-expert can understand, otherwise the ML system is just a magic black box to its users and this can cause serious ethical and practical issues.

$\endgroup$
  • $\begingroup$ Why is only considering the technical (which I assume includes statistical) a "common mistake"? The statistical theory explains why optimizing models using scores that involve thresholding is a poor decision and leads to non optimal models (which would have been found if one used a proper scoring rule). All of the reasons you give can be alleviated by optimizing a cost function related to the problem at hand (where we choose the threshold based off the tradeoff between false positives/negatives) after we find a model that has as accurate probabilities as possible. $\endgroup$ – aranglol Aug 31 at 15:49
  • $\begingroup$ I think reporting accuracy, F1 etc. if we need to for interpretation of scores is fine. But to me the theory shows that optimizing a model to these scores is dangerous. $\endgroup$ – aranglol Aug 31 at 15:53
  • $\begingroup$ @aranglol my point is about keeping in mind the final goal of the task at hand and the fact that very often whatever representation we use (in particular for optimization/evaluation) is a simplification of the task. I've seen many cases of "I have problem X, it's technically similar to problem Y so I use the same model and the same evaluation method as for Y", sometimes leading to perfectly good results according to the evaluation metric even though the metric doesn't properly represent the performance on actual task X. $\endgroup$ – Erwan Sep 1 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.