What are the advantages vs disadvantages of cross validation types? Like k-fold, leave one out, etc.
$\begingroup$
$\endgroup$
2
-
$\begingroup$ LOOCV may not work well for a data set with many samples. In the case, you would need to increase the amount you leave out $\endgroup$– GK89Sep 17, 2016 at 1:04
-
$\begingroup$ @GK89 why wont it work well? $\endgroup$– Armon SafaiSep 17, 2016 at 1:05
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
As you probably know by now, cross-validation is a method of:
- partition dataset into 'train set' and 'test set'
- fit model to train set, get the prediction error on test
- repeat (1-2) in a bunch of different, 'pre-defined' ways, storing the prediction error each time.
- average all the prediction errors to understand how your model will behave in the wild!
All the 'magic' of different cross-validation (CV) types happens at (3), basically. Say your dataset has $n$ observations.
- leave $p$ out CV separates your dataset into a train set of size $n-p$ and a test of $p$, gets the prediction error, and repeats this process for every single possible $p$-sized subset of the dataset.
- leave $1$ out CV is a special case of leave $p$ out where $p=1$
- $k$-fold cross CV partitions your dataset into $k$ equal sized partitions, trains on the first $k-1$ partitions and tests on the last partition, and gets the prediction error. The result at the end is ten $k$ prediction errors to average over. (Note that $k=n$ is the same as leave $1$ out CV.)
As for disadvantages and advantages, it's basically just a trade-off between exhaustiveness and speed. And I guess it depends on your dataset. The upper limit of exhaustiveness is basically doing leave $p$ out CV for all $p=\{1,2,...,n\}$. Probably depends too on the linearity of your dataset. In my experience, I haven't had trouble using $k$-fold cross CV.
