I have roc curve with AUC of 0.91. I applied the following function to determine the best threshold: threshold1[np.argmin(np.abs(false_positive_rate1+true_positive_rate1-1))] and I got 0.004. Does it make sense? it means that the change between the classes is very gentle, that there is not enough difference between them?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
That’s fine.
Depending on the cost of misclassification, you might choose an even lower threshold of zero. After all, many of us just spent the past two years assuming everyone had Covid-19, since the cost of a false negative is potentially so catastrophic.
The idea of the cutoff threshold being influenced by the costs of misclassifications leads to direct analysis of machine learning probability outputs via proper scoring rules. I’ll give some of my usual links on this topic.
https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/classification/ https://stats.stackexchange.com/a/359936/247274 https://stats.stackexchange.com/questions/464636/
For what it’s worth, the Frank Harrell whose blog I’ve linked has a pretty low opinion of ROC curves and would advocate for picking a threshold based on the misclassification costs (which might differ from subject to subject), not based on the ROC curve.
