1
$\begingroup$

Say I have a binary classifier. I calculate ROC to select an ideal threshold of say, 0.6. Then, I look at the AUC.

But wait! If AUC doesn't change by selecting an 0.6 threshold, then what makes AUC such a great metric?

AUC sums over all threshold values.

So my question is this:

why do we care how the model would perform as a summation over all thresholds? Don't we just ultimately want one threshold?

$\endgroup$
2

2 Answers 2

1
$\begingroup$

Roughly speaking, it depends on your purpose.

AUC is for mathematical purpose (roughly speaking). It is a characteristic of the quality of your model. It depends on your data and your skills as model-builder.

Threshold is for business purpose (roughly speaking), for making business decisions. Say, you model output is the probability of user churn. Then you need to make a business decision, which user to address in a special way (top-5% or top-10% of potential churners). And here you need to use a threshold.

$\endgroup$
0
$\begingroup$

They serve different roles. Picking a threshold of $0.6$ puts you in a position to consider a probability of $0.01$ the same label as a probability of $0.59$, but one of those missed the true label by a lot more than the other. By considering the predicted labels, rather than the probabilities, you discard that information.

If what concerns you is just the label output, then there is no need to worry about the predicted probabilities. You set yo threshold and let your AI software make categorical decisions like ringing a buzzer or paging an on-call physician. More likely, as the links on my comment describe, you are interested in the predicted probability. Then you would be concerned with a performance metric that considers the probability values. ROCAUC is such a metric, and it an example of what is known as a proper scoring rule. Even better is a strictly proper scoring rule like log loss or Brier score.

One advantage that AUC has over strictly proper scoring rules is that it is on somewhat of an absolute scale. It might be easier to see an AUC of $0.9$ and know you have a good model than to evaluate a particular log loss or Brier score. But it is the strictly proper scoring rules that pursue the true probabilities, not AUC and definitely not threshold-based metrics like accuracy, sensitivity, specificity, and $F_1$ score.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.