I have a doubt regarding the cross validation approach and train-validation-test approach.

I was told that I can split a dataset into 3 parts:

  1. Train: we train the model.
  2. Validation: we validate and adjust model parameters.
  3. Test: never seen before data. We get an unbiased final estimate.

So far, we have split into three subsets. Until here everything is okay. Attached is a picture:

enter image description here

Then I came across the K-fold cross validation approach and what I don’t understand is how I can relate the Test subset from the above approach. Meaning, in 5-fold cross validation we split the data into 5 and in each iteration the non-validation subset is used as the train subset and the validation is used as test set. But, in terms of the above mentioned example, where is the validation part in k-fold cross validation? We either have validation or test subset.

When I refer myself to train/validation/test, that “test” is the scoring:

Model development is generally a two-stage process. The first stage is training and validation, during which you apply algorithms to data for which you know the outcomes to uncover patterns between its features and the target variable. The second stage is scoring, in which you apply the trained model to a new dataset. Then, it returns outcomes in the form of probability scores for classification problems and estimated averages for regression problems. Finally, you deploy the trained model into a production application or use the insights it uncovers to improve business processes.

As an example, I found the Sci-Kit learn cross validation version as you can see in the following picture:

enter image description here

When doing the splitting, you can see that the algorithm that they give you, only takes care of the training part of the original dataset. So, in the end, we are not able to perform the Final evaluation process as you can see in the attached picture.

Thank you!



4 Answers 4


If k-fold cross-validation is used to optimize the model parameters, the training set is split into k parts. Training happens k times, each time leaving out a different part of the training set. Typically, the error of these k-models is averaged. This is done for each of the model parameters to be tested, and the model with the lowest error is chosen. The test set has not been used so far.

Only at the very end the test set is used to test the performance of the (optimized) model.

# example: k-fold cross validation for hyperparameter optimization (k=3)

original data split into training and test set:

|---------------- train ---------------------|         |--- test ---|

cross-validation: test set is not used, error is calculated from
validation set (k-times) and averaged:

|---- train ------------------|- validation -|         |--- test ---|
|---- train ---|- validation -|---- train ---|         |--- test ---|
|- validation -|----------- train -----------|         |--- test ---|

final measure of model performance: model is trained on all training data
and the error is calculated from test set:

|---------------- train ---------------------|--- test ---|

In some cases, k-fold cross-validation is used on the entire data set if no parameter optimization is needed (this is rare, but it happens). In this case there would not be a validation set and the k parts are used as a test set one by one. The error of each of these k tests is typically averaged.

# example: k-fold cross validation

|----- test -----|------------ train --------------|
|----- train ----|----- test -----|----- train ----|
|------------ train --------------|----- test -----|
  • 1
    $\begingroup$ Thank you very much for the example, Louic! This is what I was able to catch from your answer. So, initially you split the dataset into train and test. Then you put away the TEST set. Then you run CV on the training data and calculate the error based on the validation folds. That’s awesome..., here we adjust the model etc. But, when it comes to the final measure of model performance, how do you implement that? All the libraries I have been working with only provide the cross validation part but they don’t take into account the TEST split. $\endgroup$
    – NaveganTeX
    Commented May 26, 2019 at 10:19
  • 3
    $\begingroup$ In scikit-learn you can use train_test_split to split into a training and test set. After that a GridSearchCV on the training set takes care of parameter/model optimisation. Finally, you can calculate an error using the predictions on the test set (eg. using roc_auc_score, f1_score, or another appropriate error measure.Does that answer your question? $\endgroup$
    – Louic
    Commented May 26, 2019 at 17:56
  • $\begingroup$ It does indeed! I was even more confused because I’m working with recommender systems and it’s a little bit different but now I have a more clear idea. Thank you very much, Louic! $\endgroup$
    – NaveganTeX
    Commented May 26, 2019 at 21:54
  • 2
    $\begingroup$ This single answer is more informative than so many blogs on cross val that I had been exploring for days! $\endgroup$
    – Dev_Man
    Commented Oct 21, 2019 at 12:47
  • $\begingroup$ After the final testing, shouldn't we train our model one final time with the complete data, i.e. train + test together? $\endgroup$
    – dgg32
    Commented Apr 8, 2021 at 15:05

@louic's answer is correct: You split your data in two parts: training and test, and then you use k-fold cross-validation on the training dataset to tune the parameters. This is useful if you have little training data, because you don't have to exclude the validation data from the training dataset.

But I find this comment confusing: "In some cases, k-fold cross-validation is used on the entire dataset ... if no parameter optimization is needed". It's correct that if you don't need any optimization of the model after running it for the first time, the performance on the validation data from your k-fold cross-validation runs gives you an unbiased estimate of the model performance. But this is a strange case indeed. It's much more comment to use k-fold cross validation on the entire dataset, and tune your algorithm. This means you lose the unbiased estimate of the model performance, but this is not always needed.

  • 1
    $\begingroup$ Agreed, thanks. I mentioned that second example because I wanted to make it clear that cross-validation is a method (as opposed to an application of a method). To me it seemed that OP had trouble distinguishing the method and its application. $\endgroup$
    – Louic
    Commented May 26, 2019 at 11:04
  • $\begingroup$ We could use Nested Cross Validation as an alternative to cross validation + test set $\endgroup$
    – NaveganTeX
    Commented May 26, 2019 at 11:17
  • 5
    $\begingroup$ Remember that the point of k-fold cross-validation is to be able to use a good amount of test or validation data, without reducing your training dataset too much. If you have sufficient data to have a good size training data set, plus reasonable amounts of test and validation data, then I would suggest not to bother with k-fold cross validation, let alone nested solutions. $\endgroup$
    – Paul
    Commented May 26, 2019 at 11:23

@Louic is mostly right, but this thread is off on a few points:

  1. Train/val/test is a form of cross-validation. The question is comparing two methods of CV: k-fold vs "hold out". Decent resource here to read up on the various approaches.

  2. Using the test set for evaluation occurs prior to "finalizing" the model. You evaluate the model that was trained without the test set so that it is unbiased. You then discard that model and train a final model on all data (train+test+val). You consider the score you got from the test set to be an estimate of the final model. You do this because there is no other way to evaluate a model trained on all data, but this estimate is the best we can do (because the final model has more data, in theory the score would be slightly better than the estimate, but we'll never know but how much!). I would highly recommend reviewing Jason Brownlee's post on finalizing models, which I have found to be one of the few entry-level sources that's spot-on

  3. K-fold will pretty much always be better than hold-out. This is because you take advantage of the power of resampling since the test set will never be truly representative of unseen data. In other words, you would have a distribution of scores if you repeated the train+val+test process with a different mix of the same data. K-fold is the acknowledgement of this, and when allows you to measure the variance of the scores in each fold to better understand your sampling schema. Think of 5-fold CV has just doing 20% test holdout 5 times to check out good your original split was. However, if you have enough data, the gains are not worth the extra computation and therefore hold out is sufficient.

  • $\begingroup$ Thanks for your answer. Could you please check the link for Jason Brownlee's post as I get a "page not found" error when trying to follow it. $\endgroup$
    – Lynn
    Commented Jan 6, 2023 at 11:26
  • $\begingroup$ sorry. it is fixed, and is this one: machinelearningmastery.com/train-final-machine-learning-model $\endgroup$
    – pdashk
    Commented Jan 6, 2023 at 23:41

Excellent question!

I find this train/test/validation confusing (I've been doing ML for 5 years).

Who says your image is correct? Let's go to an ML authority (Sk-Learn)

In general, we do k-Fold on train/test (see Sk-Learn image below).

Technically, you could go one step further and do a Cross Validation on everything (train/test/validation). I've never done it though ...

Good luck!

enter image description here

  • 1
    $\begingroup$ We will figure it out, someday!!! $\endgroup$
    – NaveganTeX
    Commented May 26, 2019 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.