1
$\begingroup$

Goal: Compare preprocessing methods, models, and hyperparameters without leaking into the final generalization estimate, applying cross-validation (cv), i.e. NOT applying any fixed train/test splits.

Solutions found so far (and problems):

  1. "Nested cv": scores of different hyperparameters (from inner SearchCVs) are averaged in an outer cv and the best score is selected.
  • If the meaning of cv is to average scores from ONE method on different splits to get an estimate of generalization performance, this option appears as simply wronlg applying cv or am I misunderstanding this? In addition, how can I include different preprocessing methods in the search, since preprocessing defines X (has to be done before splitting X)?
  1. Successive cvs on the same entire dataset X_all. First trying different preprocessing methods, comparable models and hyperparametersets saving always the average test score. Second, using only the methods of the best score, perform a cv again on X_all only for generalization performance.
  • With this procedure, is it not easier for validation "test" information to leak into the generalization performance test? Or can this be mitigated/minimized somehow?
  1. How about a nested cv for each prepro-method loop, saving a "validation score" (result of of inner grid_search, i.e. .best_score_ with refit=True) in each fold of outer cv. In addition, saving each average test score of outer cv. Later taking the best of validation scores together with corresponding test score as estimate of generalization performance?
  • This still contains the averaging of different hyperparam combis but perhaps its better than fully leaking into the test set.
  1. The only other option appears to be fixed splits instead of cv.
$\endgroup$
2
  • $\begingroup$ I'm not sure that I understand the issue, but let's be clear about one thing: applying k-fold CV for selecting anything (e.g. model, hyperparameter,...) implies running the exact same model with the same parameters k times. Then full CV process is repeated for the next model/parameters. And indeed once the best model/parameter is selected it's the only one run on the final test set. Do you think of nested CV because for instance you select preprocessing at level 1, then model at level 2, then hyperparameters at level 3? It's not clear to me. $\endgroup$
    – Erwan
    Feb 26, 2023 at 12:35
  • $\begingroup$ Thank you @Erwan. On the first level are preprocessing methods giving rise to different sets of labeled data (X). The latter means I have to loop across prepro methods before outer cv! The second level are different model types. Third level are hyperparameters. For the latter there are inner RandSearch or GridSearchCV. But what is the best practice is for averaging validation score to get the best prepro-hyperparam-combination? And is this validation score the mean of outer fold grid_search.best_score_ with inner refit=True? How is generalization for the best combi estimated? $\endgroup$
    – le8rning
    Feb 27, 2023 at 9:23

1 Answer 1

1
$\begingroup$

(started as a comment but maybe useful as an answer)

To my knowledge, if you really want to evaluate many combinations with different levels like this, it would be nested CV with so many levels that I can't even count from your explanations.

Imho this is too complex, and it would also take a very long time to compute. The generalized performance is obtained not for a particular combination of parameters but for "following the exact same recipe" as the one which resulted in these parameters.

I can think of two alternatives:

  • Simplifying the combinations of options, i.e. instead of having multiple levels like preprocessing X then model type Y then hyperparameters Z, just have a set of predefined combinations like X1+Y1+Z2, X2+Y2+Z2,... And process these with a single CV process (not nested). Disadvantage: you have to select (and probably discard) combinations in advance.
  • Using a genetic algorithm instead of multiple nested CV to select the best conbination of options. It's more complex but it can handle a very large set of combinations and potentially converge efficiently to the optimal solution. Disadvantage: a bit complex, you often have to do some trial and error to find the right parameters for the genetic process.

Imho I would estimate the number of potential combinations to evaluate, and if it's low enough then I would choose option 1, otherwise option 2 (genetic).

$\endgroup$
9
  • $\begingroup$ Thank you again for your time and useful tips @Erwan! I am a still wondering though why my problem should be more complex than the ordinary. All time series recordings need preprocessing including feature extraction and there are many options while it is often impossible to know in advance which is best. Comparing different models and hyperparameter tuning is also regular. So the 3 levels appear the basic standard to me. Are they not? $\endgroup$
    – le8rning
    Mar 1, 2023 at 9:05
  • $\begingroup$ @le8rning I agree that this is quite common, but I would think it's quite rare that people would really need to evaluate all the options with selection like this on the same dataset. I'm not sure, but I think people wouldn't deploy a full nested CV setting for this because it becomes really complex after 2 levels. Instead they would use either the first option above, for instance pick 2 or 3 possible preprocessings and match them with all the possible models; or simply keeping a small subset of data for evaluating preprocessing, for instance. $\endgroup$
    – Erwan
    Mar 1, 2023 at 15:27
  • $\begingroup$ with your option "pick a fixed number of parameter combinations and process in a single cv", how do you process concretely? Do you apply all combinations each fold of cv (which would have the averaging across methods problem again), or do you loop across the combinations and do one cv for each (which would not be a "single cv"). Which are the actual scores you rank to select the best parameter combination? Or yet otherwise? $\endgroup$
    – le8rning
    Mar 4, 2023 at 18:09
  • $\begingroup$ @le8rning CV is a method for evaluating more reliably: for every method it's a must to run every fold, therefore you loop across the combinations and do one cv for each (it's not a single run of CV but it's a single level from the point of view of nested CV). Each CV gives an average score for one of the "full methods", including parameters. $\endgroup$
    – Erwan
    Mar 4, 2023 at 22:40
  • $\begingroup$ ok thanks for clarification @Erwan! Just to make sure: so the score you use to select the best method-combination is the mean test scores of each (outer/single level) CV, right? From which test score do you then estimate the generalization performance? Is it not a leak into the outer test using it for selection and generalization performance? $\endgroup$
    – le8rning
    Mar 6, 2023 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.