2
$\begingroup$

Confusion1) If k=10 then does this mean that 90% is for training and 10% for testing? So always we have k% for testing?

Confusion2) In the following code I have used 10-fold cross-validation for training a Support Vector Machine (SVM). In general a data set will be split into (a) Training set, meas(trainIdx,:) (b) Testing set, meas(testIdx,:) c) Validation set. In the cross-validation approach I am building the SVM learner by training and validating inside the loop. Based on my understanding, the validation data must be completely different from the training and testing. But, in many online resources it is said that after cross-validation, one must re-train on the entire data set which in this example would be the meas(:,1:end). If so, then the learned model svmModel inside the cross-validation is lost. Have I misunderstood completely wrong?

Can somebody please show what is the next step in the classification once the cross-validation is over?

$\endgroup$
1
$\begingroup$

Confusion 1)

From wikipedia :

k-fold cross validation is a technique for assessing how the results of a statistical analysis will generalize to an independent data set

They also say :

In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used,[7] but in general k remains an unfixed parameter.

So you now see that you don't have k% for the testing but you always use 1/k % of the your dataset as test set. Note : you can choose to keep 2/k or more but it will be a lot more complicated to code.

Confusion 2)

In the scikit learn they above all refer to this tool as an "evaluating performance tool" counter to what wikipedia authors may suggest. The point is that CV allows you to assess how robust and reliable your prediction will be according to your initial dataset.

The final mean obtained at the end of the CV is a mean score on the k testing sets. It is often good to have a look at all the intermediary results to assess the variances of them that can be a good explanatory estimation in case of bad generalization capability of your model.

Edit : why running another training after CV

Cross validation can also be used as an optimization tool to find the best hyper-parameters of your model. In this case, you should take the better hyper-parameters (among the k different parameters; one for each fold) and use them to do prediction on the full set to see if the optimized (i.e. choosen from CV) hyper-parameters are good on the full dataset. The notion of "best" parameter can be seen as the hyper-parameters from the model who gave the best score in your CV process.

Note you can still put aside a validation set from the dataset, on which you won't do CV. This validation set can be used as the last test of your model prediction's quality. See also here

Finally, you can use each k-fold model to predict an estimation and then take the mean of them as the final prediction of your model as Wikipedia authors have suggested, but this idea is closer to ensemble learning or a kind ofBootstrap method without replacement, than to CV

hope it helps

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you for your answer. Can you please clarify what "mean" are you suggesting to? Is the it mean of the prediction error? $\endgroup$ – Srishti M Jun 26 '18 at 15:32
  • $\begingroup$ @SrishtiM It depends on the sentences. I said "The final mean obtained at the end of the CV is a mean score on the k testing sets" here is the mean score on the test set calculated on every k-training in the CV fashion Here "k-fold model to predict an estimation and then take the mean of them as the final prediction" I mean the mean of the prediction of each of the k-models, in a prediction phase (beyond the training) $\endgroup$ – nsaura Jun 27 '18 at 8:53
  • $\begingroup$ Thank you for your answer. So, if we have 100% of the data, split it into say 80/20. Use the 80% as the training set which is again split into k folds. Inside the cv loop using this 80% data the classifier is trained and validated. To test the learned classifier, we use the 20% of the data. If the error for this test set is less than the cv error we accept the classifier and deploy it. Else, repeat the procedure. Is this the overall summary of training and testing?Did I understand you correctly? thank you very much for your time and effort. $\endgroup$ – Srishti M Jun 27 '18 at 20:17
  • $\begingroup$ Actually if you choose the k-fold Cross validation, you do it on the 100% of the dataset since the split is done by setting one fold as test case (for each of the k fold as we said earlier) See here for more information stats.stackexchange.com/questions/11602/… I will edit my answer to provide a more complete answer $\endgroup$ – nsaura Jun 27 '18 at 21:12
  • $\begingroup$ thank you for your updated answer. But what if I use Cv for training a classifier and not as an optimization tool? Then in many implementations I have seen inside the CV loop the train data is used in the learning of the model and the same train set is used for prediction. Then the error in training is calculated. Using the test set inside the cv loop, the prediction is performed and again the error is calculated using the predicted labels from the test set and the known labels for the test set. Outside the CV loop, the learned model is run on an unseen data set which is the validation set. $\endgroup$ – Srishti M Jun 27 '18 at 22:00
1
$\begingroup$

Confusion 1) If k=10 then does this mean that 90% is for training and 10% for testing? So always we have k% for testing?

No, we don't have k% for testing. It means that your data (train set) is divided into k equal parts, and one of the part is for validation set and rest other are combined to form train set. If you still have a doubt, change the value of k to any other number like 5 or 8 and after creating k folds, print the shape of train and validation set.

Also its not for testing, it is actually for validation (thats why the name cross-validation).

Confusion 2)

Personally, I have not encountered any resource that says you have to retrain on whole data. You are right, the model in cross-validation will be lost, as the parameters which the model have learnt will be replaced by new parameter values.

Your next step after cross-validation is to select the best model and proceed with that.

I think you will find a great deal of discussion here- https://stats.stackexchange.com/questions/52274/how-to-choose-a-predictive-model-after-k-fold-cross-validation

And whether to train on whole dataset or not- https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you for your answer and the links. The first link does say that we use the whole dataset after cross-validation. I am doing cross-validation to train a classifier and I thought that after the cross-validation we end up with one model. Say, I train an SVM using 10 fold CV. Then after the CV ends, I will end up with a final classifier. The second link says to use the whole data set if we are using CV to tune and estimate the hyperparameters. Then use a second CV loop to train a classifier/build a model. Therefore, both the answers are different in my opinion. $\endgroup$ – Srishti M Jun 26 '18 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.