# Does it make sense to repeat calculating AUC in logistic regression?

I have a question regarding logistic regression models and testing its skill. I am not quite sure if I understand correctly how the ROC Curve is established.

When calculating the ROC curve, is a train test split happening and then the skill of a model based on the training split is tested on the test split? or is a model based on the ENTIRE data just tested on the ENTIRE data?

If the first is the case, would it make sense to do repeated random train test splits and average out the Area under the curve? would that bring about any additional certainty about the model's skill?

Thank you.

• When calculating the ROC curve - there is no train/test split happening ! it is just a threshold that is changing - the threshold change the label of the outcome variable. unfortunately, I couldn't get more out of your question. May 27 at 10:01
• I am not sure I understand what you mean with 'Threshold change the label of the outcome variable'. So from what I understand, the ROC curve displays the True positive rate versus the False Positive rate of the model when its tested on its own data. And this RATE, is this calculated by matching the true class memberships vs the predicted class memberships?! Or am I wrong about this? I find most literature on this difficult, as the ROC is used for such a broad spectrum of things and thus the explanations of this are very broad. May 27 at 13:39
• The way a ROC curve is calculated is by using $0.01$ as the threshold. Above a probability of $0.01$, we classify as $1$; below a probability of $0.01$, we classify as $0$. Then we do the same for a threshold of $0.02$, and so on. For each threshold value, we calculate the TPR and FPR, which are what get graphed.
– Dave
May 27 at 13:42
• @DataVader your understanding of ROC and how it is calculated is wrong. I recommend watching some youtube ... also be aware the question may get downvoted by those who don't like repeated text-book questions :-) good luck. May 27 at 18:11

The calculation of ROC curve and the AUC based off of that curve is simply a comparison of the predictions from your model (logistic regression) and the actual values on some set of data. This can occur with predictions on the training set or predictions on a test set. Best practice is to do this comparison on a test set as this will best represent the performance of the model on new data.

Averaging of the AUC based on repeated train-test splits, with retraining of the logistic regression on the training set and calculation of the ROC curve and AUC on the test set can give a better estimate of the performance of the model. In addition, the distribution of the AUC across each of the splits can give a sense of the stability of the model's performance.

The repeating of train-test splits is commonly called cross-validation and most commonly performed through k-fold cross-validation where you split the data into k sets, use one as the test set and the rest as the training set. You can then repeat this process using each of the k sets as the test set.