The Goal Compare several preprocessing methods and models - while tuning hyperparameters for each model - without leaking information into the final generalization estimate, applying cross-validation (cv), not fixed train/test splits.
Not the problem: The principles of cross-validation, nested cv, fixed train/val/test splits. With fixed train/val/test splits it is easy but I thought fixed splits are too risky, or are they not (hyperparameter tuning with validation set)?
Previous attempts, proposed solutions: So far I was not able to get the precise order of procedures from previous solutions. Although 1-liner packackage solutions may be the final best practice, they are not helpful to explain what happens. Better would be pseudo-code schemes as below
https://machinelearningmastery.com/nested-cross-validation-for-machine-learning-with-python/ and https://stackoverflow.com/questions/42228735/scikit-learn-gridsearchcv-with-multiple-repetitions/42230764#42230764 In these examples, different hyperparameter searches are performed on different outer cv splits. But if the best parameter set is selected via the test score of outer splits instead of averaging these, it is not proper cv anymore, is it? If I am not mistaken, in pseudo-code, the proposed nested cv with hyperparameter tuning would look like this:
outer cv1 on X_all: for train1_ix, test_ix in cv1: Inner cv2 (GridSearch) on X_train1, proper cv, i.e. for parameter_set in parameter_grid: apply cv, average val scores compare val scores and output best parameter set collect test scores in list of outer cv splits get best hyperparameters via BEST test score in list and simultaneously average for performance estimate?
https://stackoverflow.com/questions/50148868/model-tuning-with-cross-validation: It sounds as if GridSeachCV is applied on X_all:
if without_hyperparameter_tuning: for train1_ix, test_ix in cv1: collect test score for average average test score if with_hyperparameter_tuning: "Do GridSearchCV and repeat as before". But how/on which split do you do the GridSearchCV here, on X_all? for train1_ix, test_ix in cv1: collect test score for average average test score
I also could not extract the precise order from these threads: