0
$\begingroup$

We do the following:

  1. split data_all into K folds, each consisting of data_train_k and data_test_k where k = 0, ... K-1.
  2. for each k in 0, ... K-1, split data_train_k into M folds each consisting of data_train_k,m and data_eval_k,m where m = 0, ... M-1. Then perform hyperparameter tuning via cross-validation using the M folds. We validate the optimized model using data_test_k. Now we have K tuned models with cross-validated metric, like:
k  | avg eval score on M folds | std of score on M folds | test score
0  | 0.92                      | 0.05                    | 0.93
...
K-1|0.89                       | 0.03                    | 0.88
  1. compute average test_score - this is $\hat S$, the final estimate of test score.

Is it a proper solution? Can we use it to compare different model architectures? And finally, is the estimate unbiased?

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, nested cross-validation will produce an unbiased estimate of the model building process. That is the point of it.

The whole point of this is that the inner cross-validation could be biased because you are performing a hyperparameter search on the same data you are pulling repeated validation sets from. This will likely be overly optimistic as the validation set leaks some information to the hyperparameter search process.

The typical way we counteract this is to reserve yet another set, the test set, to test on the tuned model trained on 'data_train_k'. This has drawbacks too though, because you have split your data with a randomization and so there could be some bias in the test set either way. And, you don't get bounds on the performance this way. So, if we nest the process, we can produce randomized test-sets over and over and now we can generate some unbiased bounds on our models. This is mostly for testing the whole process of building the model using your hyperparmater search technique and a particular model.

I think the architecture search needs to be part of the inner cross-validation as well. For example, if you are trying to decide between SVM vs MLP vs boosted trees, etc... then that is basically a hyperparameter as well. The outer test set needs to be completely naive to all processes. You don't want to re-run the whole nested cross-validation over and over, because now YOU are the one biasing the data by playing around with the architecture. You would now need yet another held out set to test final performance.

The final model you would use for production would of course then be trained once using all of the data and the best technique.

  • Edit: I just wanted to edit this as my last statement might be misleading. The final model can be trained using all of the data via the same technique we applied in the inner loop. It is then ready for new data and we have an idea of how it would perform on this data.
$\endgroup$
2
  • $\begingroup$ Thanks for this answer, it clarifies a lot. I am not sure about one thing though - why we don't want to select architecture as part of outer CV? I thought I understood it but now I'm confused again. What happens if I do nested CV for different types of models and then pick the one with best average (outer) test score? $\endgroup$ Mar 23, 2023 at 19:31
  • $\begingroup$ It's the same issue. If you do that, how do you know that you haven't just picked a model that is specific to the entire dataset and doesn't generalize to unseen data? GridSearchCV can accept a dictionary of different estimators at each step in the pipeline as well, so this is easy to add. $\endgroup$ Mar 23, 2023 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.