1
$\begingroup$

I have roc curve with AUC of 0.91. I applied the following function to determine the best threshold: threshold1[np.argmin(np.abs(false_positive_rate1+true_positive_rate1-1))] and I got 0.004. Does it make sense? it means that the change between the classes is very gentle, that there is not enough difference between them?

$\endgroup$

1 Answer 1

1
$\begingroup$

That’s fine.

Depending on the cost of misclassification, you might choose an even lower threshold of zero. After all, many of us just spent the past two years assuming everyone had Covid-19, since the cost of a false negative is potentially so catastrophic.

The idea of the cutoff threshold being influenced by the costs of misclassifications leads to direct analysis of machine learning probability outputs via proper scoring rules. I’ll give some of my usual links on this topic.

https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/classification/ https://stats.stackexchange.com/a/359936/247274 https://stats.stackexchange.com/questions/464636/

For what it’s worth, the Frank Harrell whose blog I’ve linked has a pretty low opinion of ROC curves and would advocate for picking a threshold based on the misclassification costs (which might differ from subject to subject), not based on the ROC curve.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.