0
$\begingroup$

I have to perform predictive model over the dataset $D$ (with 1000 obs). From $D$, I extract 700 obs for training $(T)$ and 300 obs for validation $( V )$.

I need to perform bootstrap or 10-fold cross validation sampling.

The question is which of these sets should I use?

  • Divide $D$ in 10 subsets and alternate training and validation between them ?

  • Divide $T$ (the training subset) in 10 subsets and perform training/validation on those subsets? $V$ is used only for final validation.

$\endgroup$
1
$\begingroup$

I recommend using the second option you presented. I would use $T$ with 10-fold CV to select my modeling technique and optimal tuning parameters. Take a look at what performed the best ("best" being the model that gives us the best error, but also doesn't have the error fluctuate too much from fold to fold). After selecting a model, you can use the model on $V$ to get a realistic error rate.

The reason I don't recommend the first option is: There are varying degrees of over fitting that can occur when going through model selection and model tuning, then using that same data to get an error rate. CV is a great way to limit this overfitting and it gives us a sense of performance variance which is great, but a classic hold-out validation set is the gold standard for model performance. In your case the first option might not be wrong (depends a lot on data/techniques), but if a hold-out validation set is available I would go for that.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I would prefer doing cross validation on this dataset because it removes over fitting of data.

Coming to second question it's better to do a 10-fold cross validation on your training data set(T) by diving T into 10 subsets and tuning your model accordingly. Once you are happy with model like when your preferred accuracy is reached, then test it on validation dataset. Correct me if i am wrong.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.