3
$\begingroup$

When developing a machine learning model, at what stage are ROC curve with AUC used?

Typically I have three data sets

train - validation - final test

I do K-Fold cross validation using the combined train + validation set During that phase we can calculate the metrics including true positives, false positives as well as other metrics and average them to create a plot like the ROC curve. Similar to this example from scikit-learn

However we can also get the metrics at the end by training the final model using all the data from train + validation and testing on the test set This can also give us all the metrics, classification report and ROC curve etc.

My question is, do people generally do the ROC curves twice, once during cross validation and then a second time for the final testing? OR is it something that is used only during validation phase / hyper parameter tuning when selecting the algorithm?

$\endgroup$
2
  • $\begingroup$ If you do k-fold on train+validation, what separates training from validation? What are you doing with the scores from k-fold cross-validation? $\endgroup$ – Ben Reiniger Jan 19 at 2:37
  • $\begingroup$ @BenReiniger training + validation are actually one set - they make up one data set that gets separated in each k-fold. The scores are compared with those of the test set. They can help me determine if the model is overfitting or not (i.e. if the validation scores are higher than the the scores on the test set, then the model overfit to the training/validation data. They are also used for model selection phase to choose between different models (before even using the test set to evaluate the chosen model) . $\endgroup$ – erotavlas Jan 19 at 3:09
1
$\begingroup$

The ROC-AUC curves are used to find the best threshold that optimizes True Positive Rate vs False Positive Rate. Using it in a K-Fold cross-validation is a good practice to determine the best threshold to use.

Then, your final test is here to validate that you did not overfit on some hyperparameters, including this threshold. So ROC-AUC must not be used again in final test. You should compare the results of your final test with the same threshold used in your cross-validation.

Hope it helps.

Note on threshold (EDIT):
The threshold to optimize could be the threshold to use in a binary classification problem that outputs probabilities (for instance, output of a sigmoid or a logistic regression). In that case, various threshold settings gives various the model's predictions (FPR, TPR), and so is built the ROC curve.
You could read further on sklearn guide page.

$\endgroup$
2
  • $\begingroup$ Thanks, can you clarify what is meant by threshold, and also the last sentence - 'compare the results of your final test with the same threshold used in your cross-validation' $\endgroup$ – erotavlas Jan 19 at 15:49
  • $\begingroup$ I have edited to detail the threshold part, is that more clear ? Do you need also details on the last sentence ? $\endgroup$ – etiennedm Jan 19 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.