# At what stage are ROC curves used when building machine learning model?

When developing a machine learning model, at what stage are ROC curve with AUC used?

Typically I have three data sets

train - validation - final test

I do K-Fold cross validation using the combined train + validation set During that phase we can calculate the metrics including true positives, false positives as well as other metrics and average them to create a plot like the ROC curve. Similar to this example from scikit-learn

However we can also get the metrics at the end by training the final model using all the data from train + validation and testing on the test set This can also give us all the metrics, classification report and ROC curve etc.

My question is, do people generally do the ROC curves twice, once during cross validation and then a second time for the final testing? OR is it something that is used only during validation phase / hyper parameter tuning when selecting the algorithm?

• If you do k-fold on train+validation, what separates training from validation? What are you doing with the scores from k-fold cross-validation? – Ben Reiniger Jan 19 at 2:37
• @BenReiniger training + validation are actually one set - they make up one data set that gets separated in each k-fold. The scores are compared with those of the test set. They can help me determine if the model is overfitting or not (i.e. if the validation scores are higher than the the scores on the test set, then the model overfit to the training/validation data. They are also used for model selection phase to choose between different models (before even using the test set to evaluate the chosen model) . – erotavlas Jan 19 at 3:09