0
$\begingroup$

https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation explains the procedure and the importance of doing cross-validation to assess the performance of the method/ classifier. I have few concerns which I could not clearly understand from that answer. It will be immensely helpful if these are clarified.

Consider that I am using the Matlab's fisheriris dataset. The variable meas contains 150 examples and 4 features. The varaible species contains the labels. I have put the data and labels into a variable: Data = [meas species]According to the procedure outlined above,

  • I have split the data set Data using cvpartition into 60/40 where 60% is the Xtrain and 40% is a separate Xtest data subsets.
  • Using Xtrain I perform k fold cross-valiation and inside each fold I validate the model using the indices from Xtrain. This loop is used to tune the hyperparameters of the model. I never use Xtest in selecting the hyperparameters. Is my understanding correct?

Confusion 1) The answer in the link says

You build the final model by using cross-validation on the whole set to choose the hyper-parameters and then build the classifier on the whole dataset using the optimized hyper-parameters.

"use the full dataset to produce your final model as the more data you use the more likely it is to generalise well"

I am a bit confused on what dataset and whole set are we referring to and how is building the final model by using cross-validation on the whole set using the selected hyper-parameters different from building the classifier on the whole dataset using the hyper-parameters?

I wanted to verify if my understanding of this part is correct or not. Does this statement mean that using the cross-validated hyper-parameters obtained using the Xtrain, should the classifier be build by re-training on the Xtrain subset or on Data ?

Should my final model be the one from Data?

Confusion 2) What is the role of the unseen Xtest data set? In papers is the performance reported on the Data or on the untouched Xtest ?

$\endgroup$
1
$\begingroup$

I'm not exactly sure of cvpartition's routine, so I'll try to provide a more generalised answer.

"Whole set" v "full data set"

These are the same in this instance. Model hyperparameter tuning can be done by feeding the full / whole / complete data set into a cross validation process. Inside this cross validation, the data is split into k folds, where (k - 1) folds are used to build a model, and the remaining fold is used to test the model (since the remaining data is essentially unseen to the model). This is repeated k times; results can be averaged and standard deviation calculated.

Seen v unseen data (a.k.a. modelling and hold out samples)

This is essentially a variation on the cross validation approach above, except the split is done only once: the full data is split into two subsets - Xtrain and Xtest in your case. In this approach, you would build a model using Xtrain only and test it on Xtest (there is no repetition).

So, what's the difference?

Both approaches attempt to do the same thing - somehow validate a model's performance and ability to generalise on unseen data. Cross validation is arguably the stronger technique, especially on smaller data sets.

Once you've successfully parameterised your model (using either approach), building your model on the whole data set is advisable since more data > less data.

$\endgroup$
  • $\begingroup$ Sorry to sound noisy but I have a request from you on a clarification> I did some reading and came across the book: Python Machine Learning - Second Edition By: Sebastian Raschka; Vahid Mirjalili and his blog resource sebastianraschka.com/faq/docs/evaluate-a-model.html . Is Scenario 2 the pictorial representation of the approach mentioned by you as well? $\endgroup$ – Srishti M Jul 20 '18 at 0:27
  • $\begingroup$ Then there is another approach known as the nested CV (Scenario 3) when comparing different algorithms. But many online resources suggests that nested cv is done for hyperparameter tuning by grid search inside the cv loop. However, model selection is done without using nested cv (Model Selection via K-fold Cross-validation) . In Scenario 2 the hyperparameter is tuned using k fold cross-validation as well and not nested cross-validation, which means there is only one CV loop and the grid search for tuning is performed inside this CV loop using Xtrain. $\endgroup$ – Srishti M Jul 20 '18 at 0:29
  • $\begingroup$ Shall be grateful for a clarification. Should there be a second CV loop whenever we do hyperparameter tuning Scenario 3 or is Scenario 2 the common practice. It is unclear to me whether to use one CV loop or two CV loops. $\endgroup$ – Srishti M Jul 20 '18 at 0:31
  • $\begingroup$ Scenario 2 is "vanilla" cross validation for hyperparameter tuning and is used once your modelling technique has been determined. Scenario 3 is used for model technique selection. I would suggest using whichever approach best achieves your goal. $\endgroup$ – bradS Jul 20 '18 at 16:02
  • $\begingroup$ thank you for the clarification but there is a thin line of difference I guess which I am unable to catch. Scenario 2 is used when we have decided on what learning algorithm to use for example Neural Network over SVM and then we tune the hyperparameter of the neural network by using k fold CV. Scenario 3 is used when we have not decided on the learning algorithm and apply nested CV to tune the hyperparameter + selecting the learning algorithm. Is that what you meant in your comment? $\endgroup$ – Srishti M Jul 20 '18 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.