https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation explains the procedure and the importance of doing cross-validation to assess the performance of the method/ classifier. I have few concerns which I could not clearly understand from that answer. It will be immensely helpful if these are clarified.
Consider that I am using the Matlab's
fisheriris dataset. The variable
meas contains 150 examples and 4 features. The varaible
species contains the labels. I have put the data and labels into a variable:
Data = [meas species]According to the procedure outlined above,
- I have split the data set
cvpartitioninto 60/40 where 60% is the
Xtrainand 40% is a separate
kfold cross-valiation and inside each fold I validate the model using the indices from
Xtrain. This loop is used to tune the hyperparameters of the model. I never use
Xtestin selecting the hyperparameters. Is my understanding correct?
Confusion 1) The answer in the link says
You build the final model by using cross-validation on the whole set to choose the hyper-parameters and then build the classifier on the whole dataset using the optimized hyper-parameters.
"use the full dataset to produce your final model as the more data you use the more likely it is to generalise well"
I am a bit confused on what dataset and whole set are we referring to and how is building the final model by using cross-validation on the whole set using the selected hyper-parameters different from building the classifier on the whole dataset using the hyper-parameters?
I wanted to verify if my understanding of this part is correct or not. Does this statement mean that using the cross-validated hyper-parameters obtained using the
Xtrain, should the classifier be build by re-training on the
Xtrain subset or on
Should my final model be the one from
Confusion 2) What is the role of the unseen
Xtest data set? In papers is the performance reported on the
Data or on the untouched