ROC Curve for model validation

Question

Is there a general approach that the ROC curve can be used for to validate a model? My understanding is that we can use it to compare different threshold values to determine the best, or even see how different groups behave like with k-fold validation, but that it requires to always be comparing different threshold values. I'm being told that I should be looking at using ROC curves for validating my model (logistic regression), but they do not mean looking at the threshold value of the classification; I keep getting told that it should somehow be used to validate the model in general outside of this.

The model itself doesn't even use cross-validation because the data set itself is fairly large (over a million entries in total). Am I missing something here?

Answer 1

AUC is generally a good, relatively stable metric for evaluating a binary classifier. Looking at individual ROC curves not so much. The standard definition 'area under the ROC curve' translate that you have better choices for your binarisation thresholds (for transforming the output in a binary decision). It get complex (and interesting) when the ROC of two models cross.

However, it also has some statistical interpretation (if you take a positive and a negative sample at random the model has AUC% chance of ordering the prediction correctly), so generally the higher AUC the better. However if you have a more natural metric (a natural decision threshold) you should use it because as such the ROC is a metric over all the possible thresholds.

Answer 2

The only ways that I can see how a ROC curve could be used for model validation is to check that it is above the $45$-degree line from $(0,0)$ to $(1,1)$. If the curve is below this, then then model is doing a worse job than just predicting the same value every time, regardless of the features.

Answer 3

A ROC curve has more than one use - so you're not wrong in stating that it can be used to set a prediction threshold, but your superior is probably trying to tell you that is not the reason they're interested in the ROC plot. By examining the curve, e.g. how steeply it rises on the left, how far it is from the y=x diagonal, etc., they can better assess the quality of your model. So you should include a plot of the ROC curve - to allow your superior and potentially other colleagues to gain an understanding of your model's performance in a manner they're used to.

Also be aware (as touched on in lcrmorin's answer) that ROC AUC is a useful statistic for model assessment. It is the area under the ROC curve. It is a single number which summarizes the curve and is a commonly used index of a binary classification model's performance.