0
$\begingroup$

I have a question regarding logistic regression models and testing its skill. I am not quite sure if I understand correctly how the ROC Curve is established.

When calculating the ROC curve, is a train test split happening and then the skill of a model based on the training split is tested on the test split? or is a model based on the ENTIRE data just tested on the ENTIRE data?

If the first is the case, would it make sense to do repeated random train test splits and average out the Area under the curve? would that bring about any additional certainty about the model's skill?

Thank you.

$\endgroup$
4
  • $\begingroup$ When calculating the ROC curve - there is no train/test split happening ! it is just a threshold that is changing - the threshold change the label of the outcome variable. unfortunately, I couldn't get more out of your question. $\endgroup$
    – user702846
    May 27 at 10:01
  • $\begingroup$ I am not sure I understand what you mean with 'Threshold change the label of the outcome variable'. So from what I understand, the ROC curve displays the True positive rate versus the False Positive rate of the model when its tested on its own data. And this RATE, is this calculated by matching the true class memberships vs the predicted class memberships?! Or am I wrong about this? I find most literature on this difficult, as the ROC is used for such a broad spectrum of things and thus the explanations of this are very broad. $\endgroup$
    – DataVader
    May 27 at 13:39
  • $\begingroup$ The way a ROC curve is calculated is by using $0.01$ as the threshold. Above a probability of $0.01$, we classify as $1$; below a probability of $0.01$, we classify as $0$. Then we do the same for a threshold of $0.02$, and so on. For each threshold value, we calculate the TPR and FPR, which are what get graphed. $\endgroup$
    – Dave
    May 27 at 13:42
  • $\begingroup$ @DataVader your understanding of ROC and how it is calculated is wrong. I recommend watching some youtube ... also be aware the question may get downvoted by those who don't like repeated text-book questions :-) good luck. $\endgroup$
    – user702846
    May 27 at 18:11
0
$\begingroup$

The calculation of ROC curve and the AUC based off of that curve is simply a comparison of the predictions from your model (logistic regression) and the actual values on some set of data. This can occur with predictions on the training set or predictions on a test set. Best practice is to do this comparison on a test set as this will best represent the performance of the model on new data.

Averaging of the AUC based on repeated train-test splits, with retraining of the logistic regression on the training set and calculation of the ROC curve and AUC on the test set can give a better estimate of the performance of the model. In addition, the distribution of the AUC across each of the splits can give a sense of the stability of the model's performance.

The repeating of train-test splits is commonly called cross-validation and most commonly performed through k-fold cross-validation where you split the data into k sets, use one as the test set and the rest as the training set. You can then repeat this process using each of the k sets as the test set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.