1
$\begingroup$

Is there a general approach that the ROC curve can be used for to validate a model? My understanding is that we can use it to compare different threshold values to determine the best, or even see how different groups behave like with k-fold validation, but that it requires to always be comparing different threshold values. I'm being told that I should be looking at using ROC curves for validating my model (logistic regression), but they do not mean looking at the threshold value of the classification; I keep getting told that it should somehow be used to validate the model in general outside of this.

The model itself doesn't even use cross-validation because the data set itself is fairly large (over a million entries in total). Am I missing something here?

$\endgroup$
2
  • 1
    $\begingroup$ Who said to look at the ROC curve to validate your model? What exactly did this reference say to do and in what context? $\endgroup$
    – Dave
    Nov 21, 2022 at 18:16
  • $\begingroup$ It's actually a superior of mine. I'm supposed to be showing visualizations of the model predictions and I keep getting asked to include an ROC plot. When I bring up that I can use it to help determine the classification or bias thresholds, which is no problem, I am told that it should be included for model validation. I've not been able to get a concrete explanation as to how it validates the model, and my understanding of its uses are mentioned above, so I'm a bit lost and wondering if I've missed vital information regarding them. Pressing the subject for more info has become uncomfortable. $\endgroup$
    – user143064
    Nov 21, 2022 at 18:44

3 Answers 3

2
$\begingroup$

AUC is generally a good, relatively stable metric for evaluating a binary classifier. Looking at individual ROC curves not so much. The standard definition 'area under the ROC curve' translate that you have better choices for your binarisation thresholds (for transforming the output in a binary decision). It get complex (and interesting) when the ROC of two models cross.

However, it also has some statistical interpretation (if you take a positive and a negative sample at random the model has AUC% chance of ordering the prediction correctly), so generally the higher AUC the better. However if you have a more natural metric (a natural decision threshold) you should use it because as such the ROC is a metric over all the possible thresholds.

$\endgroup$
2
$\begingroup$

The only ways that I can see how a ROC curve could be used for model validation is to check that it is above the $45$-degree line from $(0,0)$ to $(1,1)$. If the curve is below this, then then model is doing a worse job than just predicting the same value every time, regardless of the features.

$\endgroup$
0
0
$\begingroup$

A ROC curve has more than one use - so you're not wrong in stating that it can be used to set a prediction threshold, but your superior is probably trying to tell you that is not the reason they're interested in the ROC plot. By examining the curve, e.g. how steeply it rises on the left, how far it is from the y=x diagonal, etc., they can better assess the quality of your model. So you should include a plot of the ROC curve - to allow your superior and potentially other colleagues to gain an understanding of your model's performance in a manner they're used to.

Also be aware (as touched on in lcrmorin's answer) that ROC AUC is a useful statistic for model assessment. It is the area under the ROC curve. It is a single number which summarizes the curve and is a commonly used index of a binary classification model's performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.