What is the next step after k fold CV?

Question

I came across this video lecture https://www.youtube.com/watch?v=wjILv3-UGM8 on k fold cross validation (CV). The algorithm given in the video lecture is presented below:

for k = 1:5

train on all except k

get model $M_{\tilde{k}}$

calculate accuracy on $k$ as $A_k$

end

Calculate final cross validation accuracy: $A = > \frac{1}{5}\sum_{k=1}^5 A_k$

This is quite clear to me. Here $M$ is I guess just a single type of ML algorithm. However at time stamp 6:35 the presenter raises the question that what do we do with all the 5 different models that were built? According to him, we either combine all the models and make decision based on that or take the best model out of the 5. Is this statement true?

In many sites including here (https://stats.stackexchange.com/questions/310953/doubt-about-k-fold-crossvalidation?noredirect=1&lq=1 ; https://stats.stackexchange.com/questions/11602/training-on-the-full-dataset-after-cross-validation and https://stats.stackexchange.com/questions/11602/training-on-the-full-dataset-after-cross-validation) and research papers I have understood that:

-- for doing model training using k fold CV, we re-train on the entire dataset after the end of the CV loop and that is the final model.

-- We do not select any model from inside the CV loop if the idea of doing CV training is to check the accuracy of the ML algorithm on the entire dataset.

-- However, if we have multiple ML algorithms say random forest, neural network, SVM inside the CV loop then we select the algorithm with the highest accuracy.

-- Another technique, nested cross-validation is used for hyperparameter tuning.

Is my understanding correct?

Answer 1

I didn't watch the linked video but based on your explanations: yes, your understanding is correct.

A common confusion is to assume that cross-validation is similar to a regular training stage and therefore produces a model. This assumption is wrong: CV includes repeated training/testing for the purpose of evaluating the method/parameters. From this understanding it follows that:

for doing model training using k fold CV, we re-train on the entire dataset after the end of the CV loop and that is the final model.

Yes, since we want to obtain the final model as accurate as possible so we should use all the data. In this case the CV has been used to calculate a good estimate of the performance.

We do not select any model from inside the CV loop if the idea of doing CV training is to check the accuracy of the ML algorithm on the entire dataset.

Correct, otherwise there's no point using CV.

However, if we have multiple ML algorithms say random forest, neural network, SVM inside the CV loop then we select the algorithm with the highest accuracy.

Any case where multiple methods and/or parameters are being evaluated is a bit more complex than the regular case of a single method: evaluating multiple systems is by itself an additional layer of training, in the sense that we select some parameters (typically the best model) based on the data. This means that the selection itself is based on the whole data used in the CV stage, so the CV performance of the best model is akin to a performance obtained on a training set. This is why one needs another test set (or nested CV) in order to obtain the final performance of the model. An intuitive way to understand this is to imagine evaluating say millions of models with CV: the only way to know if the best performance is due to chance or not is to evaluate the corresponding model on some fresh test set.

Note: the case of combining the outputs of all the models is a different story, since this boils down to a single meta-model.

Answer 2

In the Video
I believe, in the video when it said that you have 5 models trained on 5 different datasets, it is a bit incorrect.
You have one model trained on 5 datasets. Hence you have 5 trained models.

Then it suggested to pick a model based on voting etc. This is how Ensemble models work but Cross-validation is not for the process of Ensembling the Models

Why K-Fold CV
Key goal of K-Fold CV is to provide a reliable estimate of test error with the available train data.

In a simple split approach, we might just be lucky that the validation set contains more easy examples leading to an over-optimistic evaluation of the model.
Or we might be unlucky when the validation set contains more difficult examples and the performance of the model is underestimated.
It does not rely on only one estimate of the model error, but rather on a number(K) of estimates.

Most important point to keep in mind is that you are still working on your train dataset.
With this approach, you are better assured that the score of training is the best(reliability) you can have before checking it on testing data.
Hence, you can have more trust in the Model configuration(Hyperparameter)
Since, this is still the training data, you should train the Model with the identified hyperparameters on the whole dataset.

However, if we have multiple ML algorithms say random forest, neural network, SVM inside the CV loop then we select the algorithm with the highest accuracy

I don't think we can have multiple models inside one K-Fold. If we mean repeating the k-fold on multiple models in a simple loop. Then we might pick the model with the highest score if "score" is the only criterion of evaluation.