Which of 2 options is better practice for model optimization: 1) Nested CV wrongly averaging inner CV scores. 2) Two successive CVs on X_all. Altrntv?

Question

Goal: Compare preprocessing methods, models, and hyperparameters without leaking into the final generalization estimate, applying cross-validation (cv), i.e. NOT applying any fixed train/test splits.

Solutions found so far (and problems):

1. "Nested cv": scores of different hyperparameters (from inner SearchCVs) are averaged in an outer cv and the best score is selected.

• If the meaning of cv is to average scores from ONE method on different splits to get an estimate of generalization performance, this option appears as simply wronlg applying cv or am I misunderstanding this? In addition, how can I include different preprocessing methods in the search, since preprocessing defines X (has to be done before splitting X)?

2. Successive cvs on the same entire dataset X_all. First trying different preprocessing methods, comparable models and hyperparametersets saving always the average test score. Second, using only the methods of the best score, perform a cv again on X_all only for generalization performance.

• With this procedure, is it not easier for validation "test" information to leak into the generalization performance test? Or can this be mitigated/minimized somehow?

3. How about a nested cv for each prepro-method loop, saving a "validation score" (result of of inner grid_search, i.e. .best_score_ with refit=True) in each fold of outer cv. In addition, saving each average test score of outer cv. Later taking the best of validation scores together with corresponding test score as estimate of generalization performance?

• This still contains the averaging of different hyperparam combis but perhaps its better than fully leaking into the test set.

4. The only other option appears to be fixed splits instead of cv.

Answer 1

(started as a comment but maybe useful as an answer)

To my knowledge, if you really want to evaluate many combinations with different levels like this, it would be nested CV with so many levels that I can't even count from your explanations.

Imho this is too complex, and it would also take a very long time to compute. The generalized performance is obtained not for a particular combination of parameters but for "following the exact same recipe" as the one which resulted in these parameters.

I can think of two alternatives:

• Simplifying the combinations of options, i.e. instead of having multiple levels like preprocessing X then model type Y then hyperparameters Z, just have a set of predefined combinations like X1+Y1+Z2, X2+Y2+Z2,... And process these with a single CV process (not nested). Disadvantage: you have to select (and probably discard) combinations in advance.
• Using a genetic algorithm instead of multiple nested CV to select the best conbination of options. It's more complex but it can handle a very large set of combinations and potentially converge efficiently to the optimal solution. Disadvantage: a bit complex, you often have to do some trial and error to find the right parameters for the genetic process.

Imho I would estimate the number of potential combinations to evaluate, and if it's low enough then I would choose option 1, otherwise option 2 (genetic).