1
$\begingroup$

I have recently read this:

" AUC(Area Under Curve) is good for classification problems with a class imbalance. Suppose the task is to detect dementia from speech, and 99% of people don’t have dementia and only 1% do. Then you can submit a classifier that always outputs “no dementia”, and that would achieve 99% accuracy. It would seem like your 99% accurate classifier is pretty good, when in fact it is completely useless. Using AUC scoring, your classifier would score 0.5. "

Can someone please explain why does it reach 0.5? If 99% are negative and we output always "no", wouldn't that mean that the TruePositiveRate will be very high and the FalsePositiveRate very low, resulting in an Area Under Curve close to one?

$\endgroup$
2
$\begingroup$

The AUC reaches 0.5 because of this:

$Sensitivity = \frac{True Positives}{True Positives + False Negatives}$

$Specificity = \frac{True Negatives}{True Negatives + False Negatives}$

In your case:

$Sensitivity = \frac{99%}{99%+0%}=1$

$Specificity = \frac{0%}{0%+1%}=0$

$1-Specificity = 1$

Remember that the X axis is 1-Specificity.

The point $(1,1)$ is in the diagonal line that makes the AUC equal to 0.5

If you try this exercise with any other rule like "Saying 'yes' to dementia for all patients" you will get the point $(0,0)$ and joining these points you will have the diagonal line.

$\endgroup$
  • $\begingroup$ Thank you for the clear explanation! $\endgroup$ – default_name Nov 16 at 8:09
0
$\begingroup$

As per this answer, ROC curves are better than simple accuracy for imbalanced data sets, but they are still not good. It is better to use a precision-recall curve.

$\endgroup$
  • $\begingroup$ Thank you! That is a very useful warning! $\endgroup$ – default_name Nov 22 at 23:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.