2
$\begingroup$

I'm training a model with CV and then I'm testing the predictions on a new test set.

Am I doing the right thing or is it necessary to test the predictions on a new dataset using Cross-Validation?

Thank you!

$\endgroup$
1
  • $\begingroup$ You have a wrong understanding of cross-validation. To use CV, you better separate another set from the original training set as the validation set. This is where the cross-validation is going to work. $\endgroup$ Apr 30 at 1:08
2
$\begingroup$

Everybody understands how to perform $k$-fold cross-validation but there is often quite a lot of confusion about where/how to use it. So thanks for this good question :)

First, cross-validation is a statistical method for evaluation, not for training:

  • Of course training is performed during cross-validation, but it is performed $k$ times and therefore there are $k$ models produced during the process. Each of these models is meant to be applied only to its corresponding test set, and in general it would be a mistake to use one of these models after the CV process (it would be an even bigger mistake to select the best of these $k$ models). If a final model is needed, it should be trained on the whole training set independently from the CV process.
  • The purpose of CV is to obtain a reliable evaluation of the performance of a method by minimizing the chance factor caused by splitting the data between training and test set. The variations across folds can also be used to assess the stability of the method: if there are large variations in the performance, the method is not stable and the resulting model is likely to overfit since it depends too much on the training set.

When to use a separate test set?

  • If the purpose is only to evaluate a single method reliably, then there is no need to train a final model or to evaluate it on a separate test set.
  • However quite often one uses CV to evaluate not one but multiple methods: with different algorithms, different hyper-parameters values, different features, etc. These cases fall in the category of parameter tuning, which is a kind of training: the goal is indeed to find the optimal values for some parameters, even if the parameters appear to be part of the design (e.g. some preprocessing option). Like any training stage, there is a risk of overfitting: using CV makes the risk quite low, but even with CV the risk exists (especially with a high number of methods). This means that the selection of the best method based on CV performance might be due to chance, this is why the proper methodology in this setting is to (1) select the best method according to CV performance; (2) train the final model using only this method on the full training set; (3) evaluate on a separate test set. This ensures that the performance obtained on the test set is reliable.
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.