0
$\begingroup$

I recently published a paper in which the result of a supervised model is the following. All the metrics are macro-averaged. I have been asked to comment on the gap between the AUC and the other metrics, precisely the F1.

enter image description here

Quickly this is a relatively balanced problem. 3 classes : 15%, 48%, 37%.

Thank you in advance for inputs and advice.

How would you approach justifying the results

$\endgroup$

1 Answer 1

0
$\begingroup$

The AUC (or area under the curve) measures how well a model performs compared to a random guess. For instance, an AUC of 0.5 indicates that a model does no better than a random guess. Accordingly, a score below this is poor, while a score above this indicates that the model performs better than a random guess.

The difference between AUC and the F-1 score is that the F-1 is a better diagnostic for assessing the accuracy of a model when the classes are imbalanced, e.g. many positive classes and few negative classes, or vice versa. Please see this [answer][1] for further details.

You mention that this is a "relatively balanced problem." However, if you only have a 15% representation for class 1, and 48% and 37% for classes 2 and 3 respectively, then the dataset could be imbalanced. This could indicate that the AUC is providing a falsely high accuracy reading and the F-1 score is a better gauge of the true accuracy. However, one would need to examine the data and model in further detail to be able to make this conclusion.

In this regard, one might wish to reassess whether the use of AUC is appropriate given the weights of the respective classes. It is also noteworthy that both precision and recall are lower than the AUC, accuracy, and balanced accuracy readings. Therefore, this could also cast doubt on whether AUC is a proper gauge of accuracy for this problem.

  [1]: https://stats.stackexchange.com/questions/210700/how-to-choose-between-roc-auc-and-f1-score

$\endgroup$
3
  • $\begingroup$ Thank you for the comments. In the context that we are more interested, in this multiclassification problem, in the predicted scores than the predicted outcomes, could one argue that the AUC remains the metric of interest here? $\endgroup$
    – Eric Yamga
    Commented Feb 25, 2023 at 23:03
  • $\begingroup$ To clarify, I a more interested in the knowing that the model predicted the probabilities (.15, .80., .5) if the real probabilities are (.10, .90. 0), then the model class output. $\endgroup$
    – Eric Yamga
    Commented Feb 25, 2023 at 23:07
  • 1
    $\begingroup$ You might find this useful: stephenallwright.com/f1-score-vs-auc. AUC requires predicted probabilities while F-1 requires predicted classes. In this regard, one could argue that AUC is better designed for your stated purpose. However, if the dataset is significantly imbalanced - then the discretion between AUC and F-1 is still expected to be significant. $\endgroup$ Commented Feb 25, 2023 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.