1
$\begingroup$

You split the data in k subsamples. Train it on k-1 subsamples, test it on kth subsample, record the performance with some error merric.

Do it k times for each of the k subsamples, record the error each time. Then choose the model with the lowest error? Is it the same as ensemble technique?

$\endgroup$
3
$\begingroup$

Imagine you have 1500 labeled data points, and you want to estimate how well some classifier will work on new data. One (naive) method would be to train a model with all 1500 of your data points, and then check how many of the 1500 data points were classified correctly. This is not likely to give a good estimate of the performance on new data, because new data was not used to test the model. Some models like decision trees and neural networks will often be able to get 100% accuracy on the training data, but perform much worse on new data.

So you think to yourself that you will split the data into two sets - a training set which you will build a model with, and a testing set that you will use to evaluate the model. Lets say you decided to train the model with 1000 of your examples, and evaluate with 500. This should give a reasonable estimate of how well your model will perform on new data, but it seems a bit limited; after all, one third of your data has not been used for training at all! We only have predictions for the 500 test samples - if these ones randomly happened to be easier to classify correctly on average, then our performance estimate is overly optimistic.

Cross validation is a way to address this. Lets set $k=3$, so the data is split into three sets of 500 points (A, B and C). Use A & B to train a model, and get predictions for C with this model. Use B & C to train a model, to get predictions for A. Finally, use A & C to train a model, and get predictions for B. Now we have a prediction for every point in our labeled data that came from a model trained on different data. By averaging the performance of each of these models, we can end up with a better estimate of how well the model will perform on new data.

Note that you should then re-train your model using all 1500 labeled points if you want to apply it to new data. Cross validation is only for estimating the performance of this new model. Also if your data is large enough, cross validation is probably unnecessary and you could just make a single train/test or train/valid/test split.

$\endgroup$
  • $\begingroup$ Thank you. My question is about the statement 'averaging the performance of each of these models'. In your example, we get 3 different models. Which one will we accept as final? $\endgroup$ – Victor Jan 16 '18 at 16:24
  • 1
    $\begingroup$ None of them will be the final model -- cross validation is only used to get a performance estimate. You should train a model on the full labeled data for the final model. $\endgroup$ – timleathart Jan 16 '18 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.