Which is the final model from Nested Cross Validation: Accuracy or Frequency?

Question

https://www.cnblogs.com/guo-xiang/p/8044624.html explains with a nice example the mechanics of Nested Cross Validation.

In the picture, the example shows how to use Nested CV for hyperparameter tuning of the C parameter. Tee outer CV is run for K=3 folds and 2 folds CV in the inner CV.

This is my understanding (please feel free to correct me):

Step (1) In the first fold, we have P1, P2 as the outer training fold and P3 as the outer testing fold. Using P1 & P2, the inner CV kicks off for every hyperparameter. We get 2 average accuracies: 85% for C=1 and 80% for C=2. Since the Average accuracy for C=1 is the best, we use this hyperparameter setting and re-train on P1+P2. Evaluate on the P3 outer test fold to get 89% accuracy. We save the best score = 85% and best parameter C=1.

(2) Then, from the second fold, we have P1 & P3 as the outer training fold and P2. We get 2 average accuracies. It seems from the picture that the average accuracy for C=2 is the highest in the inner CV and so C=2 is selected. Using C=2, we re-train on P1+P3 and evaluate on P2 to get 84% accuracy. We save the best score from the inner CV and best parameter C=1.

(3) Repeat the same for the last outer third fold. We get C=1 from its inner fold and re-train on P2+P3; evaluate on P1 to get 76% accuracy.

Confusion: How to select the best hyper-parameter or which is the final model? After we come out of the outer CV loop, which one is the best hyperparameter? My understanding is that C=1 corresponding to 89% accuracy obtained from the first fold in Step 1 should be the best hyperparameter selected as that gives the highest accuracy amongst all the three outer folds. OR should the decision be made on the frequency?

Answer 1

Neither.

In the usual procedure, the outer cv is not used for model selection, but rather for performance estimation. Furthermore, the "model" whose performance we are estimating is not a fixed final model, but the entire modeling pipeline, in this case the hyperparameter tuning with cross-validation inner fitting procedure. So in your diagram, the answer being provided is "83%" to the question "how well can I expect a model to perform, given that I do a 3-fold cross-validation to select between $C=1$ and $C=2$ and then refit with the best hyperparameter?". That the selected $C$ differs across different outer splits is of little concern for that question and answer.

(k-fold) cross-validation never produces a single model; usually if you want to create such a model after using CV for model/hyperparameter selection or performance estimation, you refit a final model on the entire training set. In this case, you would again perform a cross-validation for selecting $C$, but no longer nested. That would produce a final decision for which $C$ to use, and then again you would refit a single model on all of the training data (P1, P2, P3) with that $C$.

---

Among your suggestions, I think selecting hyperparameter by the best score across all folds is particularly dangerous. If one fold just happens to be much easier to predict, then the other folds won't matter for the selection, and then whichever hyperparameter performs better on this easy-to-predict fold wins, and it seems reasonably likely that hyperparameters' performances will depend on how easy the fold was to predict. Selecting instead by frequency is perhaps more robust, but it's at least not the standard approach with nested CV.

Answer 2

That example appears to be unnecessarily complex to choose between two possible hyperparameter values. There is no reason to perform nested cross validation.

There is also high variance in accuracy. In fact, there is so much variance that the hyperparameters switch rank on which one is the best. I would not trust that one hyperparameter will consistently outperform the other.