3
$\begingroup$

I am performing k-Fold Cross Validation using a Logistic Regression classifier on a dataset and computing the ROC curve and the AUC for each fold. My desired output is one ROC curve with a corresponding AUC value.

One method (taken from here) is to take the mean false positivity rates (fpr) and true positivity rates (tpr) over all folds and plot the overall ROC curve using the mean tpr and fpr values. Then compute the AUC using the mean-ROC curve. However, this method does not work well when the dataset is small. Without a long explanation, my classification is a diagnosis that uses many samples for one diagnosis and thus reduces the predictions per fold to around 3-5.

The alternative method is to save the probabilities of each prediction in every fold and then construct a ROC curve after k-Fold CV and compute the AUC using this ROC curve. However, this would mean that various models, trained on different datasets are combined into one ROC curve. I don't know if this is an issue?

What is the industry standard for model evaluation reporting when using ROC and AUC combined with k-Fold Cross validation?

-feel free to edit my question.

Improve this question
$\endgroup$
4
  • $\begingroup$ How many folds are you doing? 3-5 per fold seems very low $\endgroup$
    – TBSRounder
    Commented Oct 6, 2016 at 17:43
  • $\begingroup$ I have found a temporary solution by decreasing the number of folds from 10 to 5 - this doubled the number of diagnosis per fold and increased the resolution of the ROC, making it more sensical. $\endgroup$
    – Renier Botha
    Commented Oct 7, 2016 at 5:24
  • $\begingroup$ Nice, the standard way to do it is to find the AUC per fold, then the mean AUC would be your performance +/- sd(AUC)/sqrt(5). Your mean AUC of all the folds should be close to if you found all the out-of-fold probabilities and lined them up and found a grand AUC. Also, the most attractive benefit of using folds is to get a sense of variability in the AUC, where you might not get that in the "grand" AUC. I would also recommend looking into repeated 5-fold cross validation. Basically you do 5-fold validation multiple times to get even more AUC data points. $\endgroup$
    – TBSRounder
    Commented Oct 7, 2016 at 11:35
  • $\begingroup$ @TBSRounder, thanks for the feedback. I used that method initially, but I needed to be able to report a proper ROC curve that corresponds to the AUC calculated. When I concatenated all the probabilities from each fold to make one ROC after 5-Fold validation, the AUC from that ROC would not correspond with the mean AUC computed as you mentioned. Unfortunately, multiple 5-fold validation is not computationally viable for me right now, but thanks for the idea. Could be useful in the future for a higher resolution ROC. $\endgroup$
    – Renier Botha
    Commented Oct 7, 2016 at 12:29

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.