3
$\begingroup$

I have used a MARS model (multivariate adaptive regression splines) and I have used k fold cross validation for the evaluation of the model, obtaining the following graph:

10 ROC curves plotted together with noticeable differences, labeled by fold number

How would be the interpretation of this model? I understand that in the 6 fold, the model obtains a better AUC, but why? What is the interpretation of this? Thanks to all.

$\endgroup$
2
  • 2
    $\begingroup$ Have you looked at confidence intervals on these AUCs? Depending on how many samples you have, you may find that these AUCs are actually not statistically significantly different from each other, in which case this would just represent normal variation from the sampling process. $\endgroup$ Apr 13, 2022 at 14:17
  • 1
    $\begingroup$ Echoing the above comment, my first question would be how many samples are in each fold. If there are "quite a lot" (subjective, I know), then I might explore fold 6 and fold 8 side-by-side, to see if there is something interesting in the data. It might suggest some feature engineering, or discover some bug in data preprocessing. $\endgroup$ Apr 19, 2022 at 10:52

1 Answer 1

4
$\begingroup$

$k$-fold cross-validation simply repeats the same process with different parts of the data. Therefore any difference between different folds can only be due to chance, i.e. it's only because different instances are selected (by chance) that the results are different.

In theory, if the dataset is large and representative enough, the performance should be almost identical across different folds. Thus large differences tend to indicate that the data is not representative enough and/or that the trained model overfits (i.e. it's too complex for the data, learning detail which happen by chance instead of general patterns).

Imho this graph is a bit subjective to interpret: I would say that there are quite important variations across folds, so it could be a case of overfitting. But the different curves stay roughly around the same area, so I think it's not too bad. It's a borderline case from my point of view.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.