Training a model with Cross-Validation

Question

I'm training a model with CV and then I'm testing the predictions on a new test set.

Am I doing the right thing or is it necessary to test the predictions on a new dataset using Cross-Validation?

Thank you!

Answer 1

Everybody understands how to perform $k$-fold cross-validation but there is often quite a lot of confusion about where/how to use it. So thanks for this good question :)

First, cross-validation is a statistical method for evaluation, not for training:

• Of course training is performed during cross-validation, but it is performed $k$ times and therefore there are $k$ models produced during the process. Each of these models is meant to be applied only to its corresponding test set, and in general it would be a mistake to use one of these models after the CV process (it would be an even bigger mistake to select the best of these $k$ models). If a final model is needed, it should be trained on the whole training set independently from the CV process.
• The purpose of CV is to obtain a reliable evaluation of the performance of a method by minimizing the chance factor caused by splitting the data between training and test set. The variations across folds can also be used to assess the stability of the method: if there are large variations in the performance, the method is not stable and the resulting model is likely to overfit since it depends too much on the training set.

When to use a separate test set?

• If the purpose is only to evaluate a single method reliably, then there is no need to train a final model or to evaluate it on a separate test set.
• However quite often one uses CV to evaluate not one but multiple methods: with different algorithms, different hyper-parameters values, different features, etc. These cases fall in the category of parameter tuning, which is a kind of training: the goal is indeed to find the optimal values for some parameters, even if the parameters appear to be part of the design (e.g. some preprocessing option). Like any training stage, there is a risk of overfitting: using CV makes the risk quite low, but even with CV the risk exists (especially with a high number of methods). This means that the selection of the best method based on CV performance might be due to chance, this is why the proper methodology in this setting is to (1) select the best method according to CV performance; (2) train the final model using only this method on the full training set; (3) evaluate on a separate test set. This ensures that the performance obtained on the test set is reliable.