12
$\begingroup$

When we do k-fold cross validation, should we just use the classifier that has the highest test accuracy? What is generally the best approach in getting a classifier from cross validation?

$\endgroup$
  • $\begingroup$ I think this question depends a lot on the metric used to determine "best approach". Quantifiably many people use accuracy, but depending on the data set, other metrics can be used such as a Harmonic mean like F1-Score. So metric choice can be equally important. $\endgroup$ – Nathan McCoy Sep 13 '16 at 9:45
  • $\begingroup$ @NathanMcCoy well assuming we pick accuracy, we would choose the classifier with the highest test accuracy? $\endgroup$ – Armon Safai Sep 13 '16 at 18:26
  • $\begingroup$ Sure! Just trying to mention something else to think about :) $\endgroup$ – Nathan McCoy Sep 20 '16 at 17:39
15
$\begingroup$

You do cross-validation when you want to do any of these two things:

  • Model Selection
  • Error Estimation of a Model

Model selection can come in different scenarios:

  • Selecting one algorithm vs others for a particular problem/dataset
  • Selecting hyper-parameters of a particular algorithm for a particular problem/dataset

(please notice that if you are both selecting an algorithm - better to call it model - and also doing hyper-parameters search, you need to do Nested Cross Validation . Is Nested-CV really necessary?)

Cross-validation ensures up to some degree that the error estimate is the closest possible as generalization error for that model (although this is very hard to approximate). When observing the average error among folds you can have a good projection of the expected error for a model built on the full dataset. Also is importance to observe the variance of the prediction, this is, how much the error varies from fold to fold. If the variation is too high (considerably different values) then the model will tend to be unstable. Bootstrapping is the other method providing good approximation in this sense. I suggest to read carefully the section 7 on "Elements of Statistical Learning" Book, freely available at: ELS-Standford

As it has been mentioned before you must not take the built model in none of the folds. Instead, you have to rebuild the model with the full dataset (the one that was split into folds). If you have a separated test set, you can use it to try this final model, obtaining a similar (and must surely higher) error than the one obtained by CV. You should, however, rely on the estimated error given by the CV procedure.

After performing CV with different models (algorithm combination, etc) chose the one that performed better regarding error and its variance among folds. You will need to rebuild the model with the whole dataset. Here comes a common confusion in terms: we commongly refer to model selection, thinking that the model is the ready-to-predict model built on data, but in this case it refers to the combination of algorithm+preprocesing procedures you apply. So, to obtain the actual model you need for making predictions/classification you need to build it using the winner combination on the whole dataset.

Last thing to note is that if you are applying any kind of preprocessing the uses the class information (feature selection, LDA dimensionality reduction, etc) this must be performed in every fold, and not previously on data. This is a critical aspect. Should do the same thing if you are applying preprocessing methods that involve direct information of data (PCA, normalization, standardization, etc). You can, however, apply preprocessing that is not depend from data (deleting a variable following expert opinion, but this is kinda obvious). This video can help you in that direction: CV the right and the wrong way

Here, a final nice explanation regarding the subject: CV and model selection

$\endgroup$
  • $\begingroup$ I see, do you really need to have a separate test data or can you just train your classifier on 100% of all the data you have since we already did cross validation and got the approximate test error for the model? $\endgroup$ – Armon Safai Sep 14 '16 at 23:11
  • $\begingroup$ +1 for "chose the one that performed better regarding error and its variance among folds". Everyone seems to ignore variance among folds when selecting models. $\endgroup$ – stmax Sep 15 '16 at 5:52
  • $\begingroup$ Why should I care the test error if only the CV error is what I care? I asked because you wrote "really on the estimated error given by the CV procedure." $\endgroup$ – SmallChess Sep 15 '16 at 6:42
  • $\begingroup$ Another question. What if the best model from CV is not the best model when built for the whole dataset? $\endgroup$ – SmallChess Sep 15 '16 at 6:44
  • 1
    $\begingroup$ You build the model on the 100% of the data, this meaning on the 100% of the data used for cross validation. If you previously split original data on 80%-20% for training and further test, then you do CV on 80%, select the model and retrain on the whole 80%. Having a test set in this case (20%) just allows you to see how is behaving with new unseen data (you can think it as data that became available in the future), but the estimation of error you use is the one from CV. Hopefully it will produce a similar error with new data if it is a good model (that 20%, or actually new data). @ArmonSafai $\endgroup$ – Javierfdr Sep 15 '16 at 14:03
8
$\begingroup$

No. You don't select any of the k classifiers built during k-fold cross-validation. First of all, the purpose of cross-validation is not to come up with a predictive model, but to evaluate how accurately a predictive model will perform in practice. Second of all, for the sake of argument, let's say you were to use k-fold cross-validation with k=10 to find out which one of three different classification algorithms would be the most suitable in solving a given classification problem. In that case, the data is randomly split into k parts of equal size. One of the parts is reserved for testing and the rest k-1 parts will be used for training. The cross-validation process is repeated k (fold) times so that on every iteration different part is used for testing. After running the cross-validation you look at the results from each fold and wonder which classification algorithm (not any of the trained models!) is the most suitable. You don't want to choose the algorithm that has the highest test accuracy on one of the 10 iterations, because maybe it just happened randomly that the test data on that particular iteration contained very easy examples, which then lead to high test accuracy. What you want to do, is to choose the algorithm which produced the best accuracy averaged over all k folds. Now that you have chosen the algorithm, you can train it using your whole training data and start making predictions in the wild.

This is beyond the scope of this question, but you should also optimize model's hyperparameters (if any) to get the most out of the selected algorithm. People usually perform hyperparameter optimization using cross-validation.

$\endgroup$
  • $\begingroup$ I thought k-fold cross validation creates k different models since there are 10 different sets of training data. $\endgroup$ – Armon Safai Sep 13 '16 at 7:57
  • $\begingroup$ Check out my updated answer. I refined it to address the question more directly and to explain better with an example. $\endgroup$ – tuomastik Sep 13 '16 at 20:53
  • $\begingroup$ i see, so when you finally train your classifier, do you use ALL your data to train the classifier after you have chosen your algorithm from cross validation $\endgroup$ – Armon Safai Sep 14 '16 at 0:59
  • $\begingroup$ Yes, if you don't want to perform any further analysis, such as hyperparameter optimization, which requires a nested cross-validation where the inner CV loop takes care of hyperparameter tuning and the outer CV loop evaluates model performance on an independent dataset with optimal hyperparameters. $\endgroup$ – tuomastik Sep 16 '16 at 13:06
1
$\begingroup$

So let us assume you have training out of which you are using 80% as training and rest 20% as validation data. We can train on the 80% and test on the remaining 20% but it is possible that the 20% we took is not in resemblance with the actual testing data and might perform bad latter. So, in order to prevent this we can use k-fold cross validation.

So let us say you have different models and want to know which performs better with your dataset, k-fold cross validation works great. You can know the validation errors on the k-validation performances and choose the better model based on that. This is generally the purpose for k-fold cross validation.

Coming to just one model and if you are checking with k-fold cross-validation, you can get an approximate of errors of test data, but when you are actually training it finally, you can use the complete training data.(Because it is assumed here that the whole data will together perform better than a part of it.It might not be the case sometimes, but this is the general assumption.)

$\endgroup$
  • $\begingroup$ Sooo when we do the cross validation, we just choose the model that has the highest test accuracy? $\endgroup$ – Armon Safai Sep 13 '16 at 7:54
  • $\begingroup$ when you say test, you mean validation dataset's test right? $\endgroup$ – Hima Varsha Sep 13 '16 at 8:00
  • $\begingroup$ yea thats what i mean $\endgroup$ – Armon Safai Sep 13 '16 at 8:00
  • $\begingroup$ once we have used cross-validation to select the better performing model(for instance you have 2 models-linear regression or neural network), we train that model (whether it be the linear regression or the neural network) on all the data $\endgroup$ – Hima Varsha Sep 13 '16 at 8:03
  • 1
    $\begingroup$ @HimaVarsha so if we decide to take the average, how would we choose the "averaged classifier"? $\endgroup$ – Armon Safai Sep 13 '16 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.