3
$\begingroup$

I am performing 10-folds cross-validation to evaluate the performances of a series of models (variable selection + regression) with R. I created manually the folds with this code.

At the moment I'm performing first variable selection, then hyperparameters tuning through cv, and finally testing the performance with RMSE and MAE for all the models, but I have a doubt.

Is it correct to "use" the same fold for all the models? Or should I do a separate cv for each model?

$\endgroup$
1
$\begingroup$

I recommend trying both (more than once), and exploring any differences. In my experience, using the same set of folds for all models or using a new set of folds for each model doesn't make any material difference. Post if you find different!

Regarding "I'm performing first variable selection, then hyperparameters tuning through cv", maybe watch https://www.youtube.com/watch?reload=9&v=S06JpVoNaA0 to be sure you are not introducing any bias.

$\endgroup$
  • $\begingroup$ You were right. I tried in various ways, and I get the same results $\endgroup$ – schrodingercat Nov 8 '18 at 10:25
  • $\begingroup$ Great! While I understand the benchmarking approach, I think it is more important to be confident that cross-validation estimates are telling you the same thing from one set of folds to the next. I have found that cross-validation estimates have less variance than validation set estimates but that cross-validation estimates can still vary appreciably from one set of folds to the next. $\endgroup$ – from keras import michael Nov 8 '18 at 16:49
0
$\begingroup$

If you want to evaluate the performance of different models i.e. Model Benchmarking, it is necessary to keep the input environment same i.e. any external input like CV (number of folds).

While you can tune the model-specific parameters to optimize the model.

$\endgroup$
0
$\begingroup$

IMHO I would use the same fold for all models. First of all it can be reproducible and you are evaluating all models with the same data. So it's the same environment for benchmarking. Also you can use folds predictions for stacking.

ps: You can try to use validation set for hyperparameters tuning.

$\endgroup$
  • $\begingroup$ Thanks! About the hyperparameters tuning, at the moment I think I have too few samples (100 more or less) to set aside a validation set. $\endgroup$ – schrodingercat Oct 30 '18 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.