5
$\begingroup$

I understand very well what k-fold cross-validation is. In my studies, and at work, I've always heard something along the lines of:

We most often use k=10 because evidence shows it's the best value for k. Smaller values don't give as good estimates, and larger values don't provide much better results either.

I intuitively can wrap my head around this. However, I cannot seem to find any research that went into declaring k=10 as the default. How would one go on to prove that 10 is the best value?

I understand the effects of using a smaller k versus a larger k in terms of bias. But why is it 10? And no 5 or 20? How could one end up on the number 10?

$\endgroup$
  • 1
    $\begingroup$ stats.stackexchange.com/questions/27730/… $\endgroup$ – TitoOrt Jun 10 at 16:01
  • $\begingroup$ Thanks! Forgot to check Cross-correlated, shame on me... Though, it doesn't really show anything formal. I understand their arguments, but I feel that there still may be a more formal answer somewhere $\endgroup$ – Valentin Calomme Jun 10 at 16:16
  • 1
    $\begingroup$ I have not read the answers yet, but my original idea is that is a "random" value. Same a p-value =0.05 is used for hypothesis testing. Why not 0.04 or 0.06? I see it as a community criterion. Now I read the answers $\endgroup$ – Carlos Mougan Jun 11 at 6:50
2
$\begingroup$

Excerpt from ESL -

Ideally, if we had enough data, we would set aside a validation set and use it to assess the performance of our prediction model. Since data are often scarce, this is usually not possible. To finesse the problem, K-fold cross-validation uses part of the available data to fit the model

So, key goal is to get the variance of full data and at the same time get the validation set.

What it means -
K should be chosen in such a way that the training data has sufficient variance to enable the learning.So should be the variance in Validation data

Let's see this table for two cases. % is the portion of validation data in each case - \begin{array} {|r|r|} \hline &k=5 &k=10 &k=15 &k=20\\ \hline I\ have\ Enough\ data &\color{green}{20\%} &\color{red}{10\%} &\color{red}{6.5\%} &\color{red}{5\%}\\ \hline Not\ Enough\ Data &\color{red}{20\%} &\color{green}{10\%} &\color{red}{6.5\%} &\color{red}{5\%}\\ \hline \end{array}

Enough data case - 10% or less might not provide enough variance into the Training set.
Not Enough data - 20% Validation set might reduce the size of the Training set below the desired level.

So, I think both 20% and 10% will work for respective cases. But generally, K-Fold doesn't have much use when we have enough data. So, what remains is 10%.

Needless to say -
The best value will always a hyper-parm. But we humans prefer is to deal in a multiple of 5, 10, Or 8,16, etc. Otherwise 9,11,10,22 can be equally good.

Reference -
7.10.1 K-Fold Cross-Validation from ESL

| improve this answer | |
$\endgroup$
4
$\begingroup$

The statement:

We most often use k=10 because evidence shows it's the best value for k. Smaller values don't give as good estimates, and larger values don't provide much better results either.

Is just categorically false. The reason people default to K=10 is because they don't know how changing K effects their estimates of the generalization error and they (like you) heard somewhere along the line that K=10 was good.

To understand what makes a good value of K (and whether or not K=10 is in fact better than say K=9 or K=11) you need to understand what changing this value has on your estimate.

As K decrease the bias in your estimate increases. This is because with lower values of K you are training on less data. For example K = 2 trains on only half the data, thus you will have a pessimistic bias in your estimate since you've decreased the amount of data available for your model to learn from. K = 3 trains on two thirds of your data, more data available to train on, better performance.

It used to be thought that there was a bias/variance trade-off in that a decrease in K would cause a decrease in variance (to go along with your increased bias) and while this is partially true it does not always hold. Lower values of K will have lower variance due to the fact that your training sets are less correlated. Think of the extreme example where K = N (leave one out). All of the training sets will look extremely similar, meaning that the estimate you receive is highly dependent on the sample you have to train on. If you were to draw many samples from the population and estimate error using leave one out you would have large variance in your estimates because of the variance between samples. This was the original reasoning for believing there existed a bias/variance trade-off with the choice of K.

However, this post outlines that this is not the case and that there is no universal truth as to what happens to the variance as K increases or decreases. Some studies show the variance increases with K, some show it decreases with K.

https://stats.stackexchange.com/questions/61783/bias-and-variance-in-leave-one-out-vs-k-fold-cross-validation/357749#357749

The other thing to consider is computational complexity. If you are dealing with datasets with millions of records it may be infeasible to use a very large value of K, especially if you are doing nested and/or repeated cross validation. So many people make their choice of K based simply on time to compute.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your explanation (+1) but this is sadly not what I am looking for. I agree with everything you said and understand the effects of modifying k. What I am truly wondering is why we eventually converge on 10. Is this purely guesswork? Is there any formal explanation? $\endgroup$ – Valentin Calomme Jun 10 at 17:59
  • 1
    $\begingroup$ What I’m saying is that we don’t eventually converge on k=10. Those that are saying this likely don’t understand what they are talking about or they do and have decided it’s a nice round number that is small enough that it’s not computationally impractical while being large enough to have minimal pessimistic bias. But there is nothing special with K=10 $\endgroup$ – astel Jun 10 at 21:01
  • $\begingroup$ I am not saying there is anything magical with 10 or that we converge in some way. But clearly, there is some criterion that means that 10 is desirable, and well 9 or 11 could probably b desirable too. But are there any numbers that could be proven to be "optimal" in some way? Or is everything based on heuristics and the love of round numbers? $\endgroup$ – Valentin Calomme Jun 10 at 21:30
  • 1
    $\begingroup$ As I said, 10 is the “default” value because someone at some point said it should be a good value and someone read that and it has been repeated without knowing why. As I said, there is some benefit to a number such as ten As it is large enough that the bias is probably small but small enough that you can compute it in a reasonable amount of time. But really, the reason you think 10 is always the best is not really rooted in anything. $\endgroup$ – astel Jun 10 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.