Why is the k-fold cross-validation needed?

Question

I am using k-fold cross-validation but do not understand it's aim. Before splitting the data set in training and test data set, one usually randomizes the entries of the data set. Given the training data set, the k-fold cross validation is done with the purpose of estimating beforehand how well the model would perform. Given the randomization, it is unlikely that there will be a dramatic change from one run into the next one in the loop of the cross-validation. At most one observes some fluctuation which is normal.

• Given the performances inside the cross-validation, what question does the cross validation answer ? Assuming that the performance is low, does it mean that we should abandon training the model ?
• I could understand the aim of the k-fold cross validation, if training the whole model is expensive in time or other resources. In that scenario splitting the training set in k subsets on which the validation is done, could potentially provide a forecast for the performance of the model. Given that even on big data, fitting the model on the entire training set is done with acceptable time complexity, I thus do not understand the role of cross-validation.
• In case we use many estimators on the same data set aiming to pick the best one, eg. SVM, logistic regression, deep learning, do one has to perform the k-fold cross-validation on each one of them ? Can one expect that the performance of the k-fold cross-validation will be similar along the different estimators ? What is the best strategy ?

Thanks for any answer.

Answer 1

Given the randomization, it is unlikely that there will be a dramatic change from one run into the next one in the loop of the cross-validation.

This assumption is wrong, it's true only if the training data is large and representative enough with respect to the features, the target class and the complexity of the model. The test set must also be large enough. In other words, if the model overfits, i.e. captures patterns which happen by chance in the current training set, the performance is likely to differ across the runs. Of course this can also happen if the current test set is small (for instance doing 10-fold CV with 100 instances means that the test set for a run contains only 10 instances).

Given the performances inside the cross-validation, what question does the cross validation answer ? Assuming that the performance is low, does it mean that we should abandon training the model ?

Cross-validation provides a statistically more reliable estimation of the performance than a single training/test set split. A simple way to observe this is to train many different models where some hyper-parameters vary. It's likely that the best combination of parameters according to a single training/test set split would be different from the best obtained by averaging across the k-fold CV runs. This is because with a sufficiently high number of models, it becomes very likely that one of them happens to perform better on a particular test set by chance. Averaging across several runs obtained on different test sets reduces this risk.

Beyond performance evaluation, CV also helps diagnose other problems which are not visible with a single training/test set split thanks to the variations across runs.

"performance is low" is interpretative, it depends on the context. Anyway the decision to abandon a project or not usually depends on many factors. But in some cases one might want to reconsider the design and/or method if performance is not satisfactory, this is not specific to CV.

I could understand the aim of the k-fold cross validation, if training the whole model is expensive in time or other resources. In that scenario splitting the training set in k subsets on which the validation is done, could potentially provide a forecast for the performance of the model. Given that even on big data, fitting the model on the entire training set is done with acceptable time complexity, I thus do not understand the role of cross-validation.

This reasoning seems illogical to me, given that $k$-fold CV:

• is only meant to provide an estimation of performance, not to train the final model.
• requires $k$ times more resources than a single train/test split.

It's the opposite: although it's more resource-intensive, CV is sometimes worth running for the advantages explained above. It's not a necessary step though.

In case we use many estimators on the same data set aiming to pick the best one, eg. SVM, logistic regression, deep learning, do one has to perform the k-fold cross-validation on each one of them ? Can one expect that the performance of the k-fold cross-validation will be similar along the different estimators ? What is the best strategy ?

This is an interesting point and a frequent source of confusion. First, no, the performance of the different models is not necessarily similar.

• Any form of selecting an optimal model based on performance, e.g. varying hyper-parameters, subsets of features or learning algorithm, is by itself a training stage. This implies that the performance obtained by the best model is not (necessarily) its true performance, so it must be evaluated on a fresh test set.
• Technically this doesn't mean that CV must be used, it just means that the data must be split three way:
  • The training set is used to train all the models
  • The validation set is used to evaluate all the models
  • The best model is selected and only this model is evaluated on the final test set.
• But very often one uses CV instead of training+validation in the first step, this way the performance of all the models is evaluated more reliably. This has the additional advantage that CV takes care of the splitting, so in this case:
  • Use CV to evaluate all the models on the training set.
  • The best model is selected and only this model is evaluated on the test set.

Answer 2

Cross validation is one way of testing models (actually very similar to having a test set). Often you need to tune hyperparameter to optimize models. In this case tuning the model with cross validation (on the train set) is very helpful. Here you do not need to use the test set (so you don‘t risk leakage). The test set still will be available for final model comparison.

Cross validation has one important advantage over using a test set only (especially with limited amount of data). A test set will only contain a small share of the data (randomly picked). In this case, it can happen, that the test set does not very well represent „new unseen data“ and thus using a test set only may lead to somewhat optimistic estimates of the true performance (at least there can be quite some variations in test results). Using cross validation, you are able to use all of the available data for the purpose of testing. So you can make sure that „bad“ observations also have a say in the process of testing.

Have a look at ISL, Chapter 5 for a sound introduction: https://hastie.su.domains/ISLR2/ISLRv2_website.pdf