1
$\begingroup$

I fully understand the data partition in a nested k-fold CV. But reading this:

Within each outer fold, the best performing model was selected based on mean root mean squared error (RMSE) over the inner folds. The model was then retrained on all training and validation data from the inner folds and final generalization performance was evaluated on the held-out test data of the outer fold. Repeating this process for each outer fold yielded 3 best-performing models, and the mean test performance of these models is reported here...

I have a question: after the inner loop is complete, they've retrained the best model on the whole inner dataset (which makes sense, and is permittable since the tuning process hasn't "seen" the outer, test dataset) but in this retraining, which model over training epochs is selected now?

I believe there are two options: You'd treat the new test data as validation data and pick the best validation accuracy epoch during training,

or you can pick the best training accuracy epoch during training and test it independently on the test data.

Which one is it?

$\endgroup$

1 Answer 1

1
$\begingroup$

The model was then retrained on all training and validation data from the inner folds

My understanding is that the inner fold already contains some training and also validation data, so retraining is done with the latter as validation data.

There is another reason not to use the new outer fold test data as validation: this causes a bias in the obtained performance, unless there is some other fresh test data on which the model is evaluated later.

$\endgroup$
3
  • $\begingroup$ Sorry but your answer is a bit ambiguous for me, so the correct answer would be to "pick the best training accuracy epoch during training and test it independently on the test data"? $\endgroup$ Sep 15, 2023 at 12:14
  • $\begingroup$ Without retraining, the answer would be choosing the model with highest validation accuracy (which happens in inner loop). That is: we pick the model on a epoch with highest validation accuracy. Then test it on the outer loop data, test set. But what to do with retraining? $\endgroup$ Sep 15, 2023 at 12:17
  • 1
    $\begingroup$ @AmirhosseinRezaei Yes, I think that the correct answer is to "pick the best training accuracy epoch during training and test it independently on the test data". About your 2nd question, I assume that the nested CV is intended to select some set of hyper-parameters which differs from the regular weights, otherwise there would be little reason to run this costly process. Thus imho 1) the regular training selects the optimal weights 2) the inner CV selects the best hyper-parameters 3) the outer CV evaluates the final model. For this part I'm guessing a bit, since I don't have the paper. $\endgroup$
    – Erwan
    Sep 15, 2023 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.