3
$\begingroup$

https://www.cnblogs.com/guo-xiang/p/8044624.html explains with a nice example the mechanics of Nested Cross Validation.

img In the picture, the example shows how to use Nested CV for hyperparameter tuning of the C parameter. Tee outer CV is run for K=3 folds and 2 folds CV in the inner CV.

This is my understanding (please feel free to correct me):

Step (1) In the first fold, we have P1, P2 as the outer training fold and P3 as the outer testing fold. Using P1 & P2, the inner CV kicks off for every hyperparameter. We get 2 average accuracies: 85% for C=1 and 80% for C=2. Since the Average accuracy for C=1 is the best, we use this hyperparameter setting and re-train on P1+P2. Evaluate on the P3 outer test fold to get 89% accuracy. We save the best score = 85% and best parameter C=1.

(2) Then, from the second fold, we have P1 & P3 as the outer training fold and P2. We get 2 average accuracies. It seems from the picture that the average accuracy for C=2 is the highest in the inner CV and so C=2 is selected. Using C=2, we re-train on P1+P3 and evaluate on P2 to get 84% accuracy. We save the best score from the inner CV and best parameter C=1.

(3) Repeat the same for the last outer third fold. We get C=1 from its inner fold and re-train on P2+P3; evaluate on P1 to get 76% accuracy.

Confusion: How to select the best hyper-parameter or which is the final model? After we come out of the outer CV loop, which one is the best hyperparameter? My understanding is that C=1 corresponding to 89% accuracy obtained from the first fold in Step 1 should be the best hyperparameter selected as that gives the highest accuracy amongst all the three outer folds. OR should the decision be made on the frequency?

$\endgroup$

2 Answers 2

3
$\begingroup$

Neither.

In the usual procedure, the outer cv is not used for model selection, but rather for performance estimation. Furthermore, the "model" whose performance we are estimating is not a fixed final model, but the entire modeling pipeline, in this case the hyperparameter tuning with cross-validation inner fitting procedure. So in your diagram, the answer being provided is "83%" to the question "how well can I expect a model to perform, given that I do a 3-fold cross-validation to select between $C=1$ and $C=2$ and then refit with the best hyperparameter?". That the selected $C$ differs across different outer splits is of little concern for that question and answer.

(k-fold) cross-validation never produces a single model; usually if you want to create such a model after using CV for model/hyperparameter selection or performance estimation, you refit a final model on the entire training set. In this case, you would again perform a cross-validation for selecting $C$, but no longer nested. That would produce a final decision for which $C$ to use, and then again you would refit a single model on all of the training data (P1, P2, P3) with that $C$.


Among your suggestions, I think selecting hyperparameter by the best score across all folds is particularly dangerous. If one fold just happens to be much easier to predict, then the other folds won't matter for the selection, and then whichever hyperparameter performs better on this easy-to-predict fold wins, and it seems reasonably likely that hyperparameters' performances will depend on how easy the fold was to predict. Selecting instead by frequency is perhaps more robust, but it's at least not the standard approach with nested CV.

$\endgroup$
8
  • $\begingroup$ Thank you for answering. Could you please clarify (a) How 83% comes (b) are you suggesting that selecting the hyperparameter based on the frequency is the correct common approach? It is not clear and I'm a bit confused. $\endgroup$
    – Sm1
    Nov 29, 2022 at 21:06
  • $\begingroup$ @Sm1 The 83% is the top-right number in your graphic, the average out-of-fold score achieved in the outer cv (the average of 89, 84, 76). My last sentence came the closest to answering (b), but I've tried to clarify the answer: the standard approach of nested CV doesn't choose (final) hyperparameters. $\endgroup$
    – Ben Reiniger
    Nov 29, 2022 at 21:13
  • 1
    $\begingroup$ @BenReiniger Very clear answer! Should we follow the same approach when we perform vanilla (non-nested) CV to tune parameters and use a test set to estimate the performance? Suppose that we are given a dataset. 1). Split into train and test set. 2). Tune parameters in train with CV and select the best one, call it A. 3). Estimate the performance of the learning algorithm on test set. In this step we estimate the performace of the learning pipeline (internally performing CV). 4). Perform again CV on the whole dataset. 5). Find best hyperparams B and refit on the whole dataset. $\endgroup$
    – ado sar
    Oct 4, 2023 at 8:39
  • 1
    $\begingroup$ I wanted to add this comment, because usually ones just refits the best configuration after step 3, in the whole dataset, omitting points 4 and 5. Is this correct or we should again CV on the whole dataset? $\endgroup$
    – ado sar
    Oct 4, 2023 at 8:41
  • 1
    $\begingroup$ @adosar That's a great question, and one I've thought about a little but not gotten around to playing around with. I suspect "the correct" answer is to run the hyperparameter tuner on the whole dataset, possibly surfacing different optimal hyperparameters, your 4-5. But I also suspect skipping them won't impact the performance very much, unless some hyperparameters are very dependent on dataset size, or your test set is somehow quite different than the training set. $\endgroup$
    – Ben Reiniger
    Oct 4, 2023 at 14:41
0
$\begingroup$

That example appears to be unnecessarily complex to choose between two possible hyperparameter values. There is no reason to perform nested cross validation.

There is also high variance in accuracy. In fact, there is so much variance that the hyperparameters switch rank on which one is the best. I would not trust that one hyperparameter will consistently outperform the other.

$\endgroup$
1
  • 2
    $\begingroup$ Thank you for answering. If we ignore the example and the number of hyperparameters used for illustrating, then in general is my understanding of the nested CV approach correct and what is the final model chosen? $\endgroup$
    – Sm1
    Nov 22, 2022 at 2:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.