Terminology - cross-validation, testing and validation set for classification task

Question

Confusion1) If k=10 then does this mean that 90% is for training and 10% for testing? So always we have k% for testing?

Confusion2) In the following code I have used 10-fold cross-validation for training a Support Vector Machine (SVM). In general a data set will be split into (a) Training set, meas(trainIdx,:) (b) Testing set, meas(testIdx,:) c) Validation set. In the cross-validation approach I am building the SVM learner by training and validating inside the loop. Based on my understanding, the validation data must be completely different from the training and testing. But, in many online resources it is said that after cross-validation, one must re-train on the entire data set which in this example would be the meas(:,1:end). If so, then the learned model svmModel inside the cross-validation is lost. Have I misunderstood completely wrong?

Can somebody please show what is the next step in the classification once the cross-validation is over?

Answer 1

Confusion 1)

From wikipedia :

k-fold cross validation is a technique for assessing how the results of a statistical analysis will generalize to an independent data set

They also say :

In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used,[7] but in general k remains an unfixed parameter.

So you now see that you don't have k% for the testing but you always use 1/k % of the your dataset as test set. Note : you can choose to keep 2/k or more but it will be a lot more complicated to code.

Confusion 2)

In the scikit learn they above all refer to this tool as an "evaluating performance tool" counter to what wikipedia authors may suggest. The point is that CV allows you to assess how robust and reliable your prediction will be according to your initial dataset.

The final mean obtained at the end of the CV is a mean score on the k testing sets. It is often good to have a look at all the intermediary results to assess the variances of them that can be a good explanatory estimation in case of bad generalization capability of your model.

Edit : why running another training after CV

Cross validation can also be used as an optimization tool to find the best hyper-parameters of your model. In this case, you should take the better hyper-parameters (among the k different parameters; one for each fold) and use them to do prediction on the full set to see if the optimized (i.e. choosen from CV) hyper-parameters are good on the full dataset. The notion of "best" parameter can be seen as the hyper-parameters from the model who gave the best score in your CV process.

Note you can still put aside a validation set from the dataset, on which you won't do CV. This validation set can be used as the last test of your model prediction's quality. See also here

Finally, you can use each k-fold model to predict an estimation and then take the mean of them as the final prediction of your model as Wikipedia authors have suggested, but this idea is closer to ensemble learning or a kind ofBootstrap method without replacement, than to CV

hope it helps

Answer 2

Confusion 1) If k=10 then does this mean that 90% is for training and 10% for testing? So always we have k% for testing?

No, we don't have k% for testing. It means that your data (train set) is divided into k equal parts, and one of the part is for validation set and rest other are combined to form train set. If you still have a doubt, change the value of k to any other number like 5 or 8 and after creating k folds, print the shape of train and validation set.

Also its not for testing, it is actually for validation (thats why the name cross-validation).

Confusion 2)

Personally, I have not encountered any resource that says you have to retrain on whole data. You are right, the model in cross-validation will be lost, as the parameters which the model have learnt will be replaced by new parameter values.

Your next step after cross-validation is to select the best model and proceed with that.

I think you will find a great deal of discussion here- https://stats.stackexchange.com/questions/52274/how-to-choose-a-predictive-model-after-k-fold-cross-validation

And whether to train on whole dataset or not- https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation