When developing a machine learning model, at what stage are ROC curve with AUC used?
Typically I have three data sets
I do K-Fold cross validation using the combined
train + validation set
During that phase we can calculate the metrics including true positives, false positives as well as other metrics and average them to create a plot like the ROC curve. Similar to this example from scikit-learn
However we can also get the metrics at the end by training the final model using all the data from
train + validation and testing on the
test set This can also give us all the metrics, classification report and ROC curve etc.
My question is, do people generally do the ROC curves twice, once during cross validation and then a second time for the final testing? OR is it something that is used only during validation phase / hyper parameter tuning when selecting the algorithm?