I am performing 10-folds cross-validation to evaluate the performances of a series of models (variable selection + regression) with R. I created manually the folds with this code.
At the moment I'm performing first variable selection, then hyperparameters tuning through cv, and finally testing the performance with RMSE and MAE for all the models, but I have a doubt.
Is it correct to "use" the same fold for all the models? Or should I do a separate cv for each model?
I recommend trying both (more than once), and exploring any differences. In my experience, using the same set of folds for all models or using a new set of folds for each model doesn't make any material difference. Post if you find different!
Regarding "I'm performing first variable selection, then hyperparameters tuning through cv", maybe watch https://www.youtube.com/watch?reload=9&v=S06JpVoNaA0 to be sure you are not introducing any bias.
If you want to evaluate the performance of different models i.e. Model Benchmarking, it is necessary to keep the input environment same i.e. any external input like CV (number of folds).
While you can tune the model-specific parameters to optimize the model.
IMHO I would use the same fold for all models. First of all it can be reproducible and you are evaluating all models with the same data. So it's the same environment for benchmarking. Also you can use folds predictions for stacking.
ps: You can try to use validation set for hyperparameters tuning.
