1
$\begingroup$

I stumbled upon a 3-class classification problem where all compared classifiers yield a higher AUC than accuracy (usually around 10% higher). This happens both when the dataset is balanced or slightly imbalanced.

Now, after looking at this answer: Why is AUC higher for a classifier that is less accurate than for one that is more accurate? I understand that, for binary classification, this might happen because the accuracy is typically computed at a threshold of 0.5 whereas AUC is based on all threshold values.

But what happens with multi-class classification? Specifically, those scenarios where accuracy is defined as the frequency with which the predicted labels match the true labels (tf.keras.metrics.CategoricalAccuracy) and AUC is defined as the weighted average of the AUC for each class vs the rest (One-vs-rest) (sklearn.roc_auc_score). Why might AUC be higher there?

In other words, I'm trying to understand what this result means. Does it mean that my classifiers can discern well when each class is measured against the others (AUC), but not as well when the prediction probabilities are an output of the softmax function, and therefore spread out for the three classes?

$\endgroup$
1
$\begingroup$

I think the question can se splitted to three parts. The first part is about comparing accuracy and AUC of the same model, the second is about comparing models and the third is about multi-class problems.

First part - I think that accuracy and AUC are not comparable metrics, I don't think that there is special meaning to the case where one is higher than the other.

Second part - ROC plot and AUC are good for understanding model and choosing model types, but in the end of the day you usually want to evaluate your final model with specific threshold so accuracy, precision and recall are the relevant metrics.

Third part - I think the answers to the previous parts above are valid both for binary classification and multi-class classification. Why do you think it have different meaning here?

$\endgroup$
2
  • 1
    $\begingroup$ Thank you, unfortunately I cannot upvote your answer as I do not have a high enough reputation. For the third part: I am confused because there is no "threshold" for categorical accuracy, there is only the class with the highest predicted probability. I can understand that a 0.5 threshold is not optimal, but what about that case? Does that mean I could choose to pick class A only when its probability is higher than the one for class B and C by a certain threshold? $\endgroup$ – razumichin May 28 at 12:01
  • 1
    $\begingroup$ You are right, there is not threshold in multi-class classification models so it is a different case, I didn't think about it. With saying that, I still think my answers are valid. I'll expalin - one model can be excellent on areas of the ROC curves which are less relevant in real-world situations and therefore it will get high AUC score, but another model will be good only on relevant areas of the ROC so it will have the higher accuracy. $\endgroup$ – Amit Keinan May 28 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.