2
$\begingroup$

What are the advantages vs disadvantages of cross validation types? Like k-fold, leave one out, etc.

$\endgroup$
  • $\begingroup$ LOOCV may not work well for a data set with many samples. In the case, you would need to increase the amount you leave out $\endgroup$ – GK89 Sep 17 '16 at 1:04
  • $\begingroup$ @GK89 why wont it work well? $\endgroup$ – Armon Safai Sep 17 '16 at 1:05
4
$\begingroup$

As you probably know by now, cross-validation is a method of:

  1. partition dataset into 'train set' and 'test set'
  2. fit model to train set, get the prediction error on test
  3. repeat (1-2) in a bunch of different, 'pre-defined' ways, storing the prediction error each time.
  4. average all the prediction errors to understand how your model will behave in the wild!

All the 'magic' of different cross-validation (CV) types happens at (3), basically. Say your dataset has $n$ observations.

  • leave $p$ out CV separates your dataset into a train set of size $n-p$ and a test of $p$, gets the prediction error, and repeats this process for every single possible $p$-sized subset of the dataset.
  • leave $1$ out CV is a special case of leave $p$ out where $p=1$
  • $k$-fold cross CV partitions your dataset into $k$ equal sized partitions, trains on the first $k-1$ partitions and tests on the last partition, and gets the prediction error. The result at the end is ten $k$ prediction errors to average over. (Note that $k=n$ is the same as leave $1$ out CV.)

As for disadvantages and advantages, it's basically just a trade-off between exhaustiveness and speed. And I guess it depends on your dataset. The upper limit of exhaustiveness is basically doing leave $p$ out CV for all $p=\{1,2,...,n\}$. Probably depends too on the linearity of your dataset. In my experience, I haven't had trouble using $k$-fold cross CV.

Reference

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.