Does double cross-validation make sense?

Question

We do the following:

1. split data_all into K folds, each consisting of data_train_k and data_test_k where k = 0, ... K-1.
2. for each k in 0, ... K-1, split data_train_k into M folds each consisting of data_train_k,m and data_eval_k,m where m = 0, ... M-1. Then perform hyperparameter tuning via cross-validation using the M folds. We validate the optimized model using data_test_k. Now we have K tuned models with cross-validated metric, like:

k  | avg eval score on M folds | std of score on M folds | test score
0  | 0.92                      | 0.05                    | 0.93
...
K-1|0.89                       | 0.03                    | 0.88

3. compute average test_score - this is $\hat S$, the final estimate of test score.

Is it a proper solution? Can we use it to compare different model architectures? And finally, is the estimate unbiased?

Answer 1

Yes, nested cross-validation will produce an unbiased estimate of the model building process. That is the point of it.

The whole point of this is that the inner cross-validation could be biased because you are performing a hyperparameter search on the same data you are pulling repeated validation sets from. This will likely be overly optimistic as the validation set leaks some information to the hyperparameter search process.

The typical way we counteract this is to reserve yet another set, the test set, to test on the tuned model trained on 'data_train_k'. This has drawbacks too though, because you have split your data with a randomization and so there could be some bias in the test set either way. And, you don't get bounds on the performance this way. So, if we nest the process, we can produce randomized test-sets over and over and now we can generate some unbiased bounds on our models. This is mostly for testing the whole process of building the model using your hyperparmater search technique and a particular model.

I think the architecture search needs to be part of the inner cross-validation as well. For example, if you are trying to decide between SVM vs MLP vs boosted trees, etc... then that is basically a hyperparameter as well. The outer test set needs to be completely naive to all processes. You don't want to re-run the whole nested cross-validation over and over, because now YOU are the one biasing the data by playing around with the architecture. You would now need yet another held out set to test final performance.

The final model you would use for production would of course then be trained once using all of the data and the best technique.

• Edit: I just wanted to edit this as my last statement might be misleading. The final model can be trained using all of the data via the same technique we applied in the inner loop. It is then ready for new data and we have an idea of how it would perform on this data.