4
$\begingroup$

Can someone explain why increasing the number of folds in a cross validation increases the variation (or the standard deviation) of the scores in each fold.

I've logged the data below. I'm working on the Titanic dataset and there is around 800 instances. I'm using a StratifiedKFold and accuracy scoring metric.

I thought that adding more data decreased variance - so if my understanding is correct, adding more folds would increase the amount of data supplied to each fit? But it appears that the more folds and LESS data passed in the lower the Standard Deviation (but the mean accuracy for each CV remains around the same)

{5: {'Mean': 0.8136965664427847, 'Std': 0.015594305964595902},
 15: {'Mean': 0.8239359698681732, 'Std': 0.0394725492730379},
 25: {'Mean': 0.823968253968254, 'Std': 0.07380525674642965},
 35: {'Mean': 0.8284835164835165, 'Std': 0.08302266965043076},
 45: {'Mean': 0.8207602339181288, 'Std': 0.09361950295425485},
 55: {'Mean': 0.8243315508021392, 'Std': 0.08561359961087428},
 65: {'Mean': 0.8273034657650041, 'Std': 0.10483277787806128},
 75: {'Mean': 0.8274747474747474, 'Std': 0.11745811393744522},
 85: {'Mean': 0.8240641711229945, 'Std': 0.12444299530668741},
 95: {'Mean': 0.8305263157894738, 'Std': 0.12484655607120225},
 105: {'Mean': 0.8243386243386243, 'Std': 0.1399822172135676},
 115: {'Mean': 0.8240683229813665, 'Std': 0.12916193497823075},
 125: {'Mean': 0.8249999999999998, 'Std': 0.13334396216138908},
 135: {'Mean': 0.8306878306878307, 'Std': 0.15391278842405914},
 145: {'Mean': 0.8272577996715927, 'Std': 0.1552827992878498},
 155: {'Mean': 0.8240860215053764, 'Std': 0.16756897617377703},
 165: {'Mean': 0.8270707070707071, 'Std': 0.16212344628562209},
 175: {'Mean': 0.824, 'Std': 0.16293498557341674},
 185: {'Mean': 0.8278378378378377, 'Std': 0.1664272446370702},
 195: {'Mean': 0.8284615384615385, 'Std': 0.17533175091718106},
 205: {'Mean': 0.8265853658536585, 'Std': 0.185808841661263},
 215: {'Mean': 0.8265116279069767, 'Std': 0.188431515175417},
 225: {'Mean': 0.8288888888888889, 'Std': 0.17685175489623095},
 235: {'Mean': 0.8294326241134752, 'Std': 0.19467536066874633},
 245: {'Mean': 0.8231292517006802, 'Std': 0.2009280149561644},
 255: {'Mean': 0.823202614379085, 'Std': 0.20790684270535614},
 265: {'Mean': 0.8254716981132075, 'Std': 0.2109826210610222},
 275: {'Mean': 0.8254545454545454, 'Std': 0.2144726806895627},
 285: {'Mean': 0.8242690058479532, 'Std': 0.2182928219064767},
 295: {'Mean': 0.823728813559322, 'Std': 0.22096355056065273}}

Folds

$\endgroup$
5
$\begingroup$

This has been a discussion for quite a while. For a more theoretical point of view you can find a good summary here.

From a practical point of view I'd look at it as follows. With increasing $k$ two things happen:

  1. Your $k-1$ training folds increase in size
  2. Your validation folds decreases in size

From the first point you can draw the conclusion that your $k$ models become more similar since your training data becomes more similar since you splitt off less data for the validation sets in the $k$-th fold. Which might lead to less between model-variance.

Moreover, since you are training on more data, the relative model complexity (i.e. compared to your data) decreases. In the bias-variance-trade-off graph (taken from [1])

Bias-variance-trade-off

this means we are moving towards the left, i.e. we trade less model variance for more model bias (please note that model variance here has a more general and conceptual meaning than the calculated variance or standard deviation between folds). The reason is that we are fitting a model with constant complexity to more data as $k$ increases, i.e. it becomes harder to learn the training data.

However, we are not only increasing the size of our training set with increasing $k$. We are also decreasing the validation set in size (see point two from above). Therefore, there might be higher between-fold variance with regards to our validation sets. This might be more relevant if the overall dataset carries higher variance or more outliers.


[1] "The Elements of Statistical Learning" by Hastie et al

| improve this answer | |
$\endgroup$
3
$\begingroup$

As your test set in the Cross-validation becomes smaller the variation becomes bigger.

Get to the limit case when you do Leave One Out Cross Validation (LOOCV), there will be one instance in the test set. Some instances will have a good performance while others will be really bad. The variation will be high.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks Carlos, you're back suggesting LOOCV again! :) Can you explain why there is more variance? I dont understand this part. I thought giving more data will help decrease the variance, so the SD of scores should decrease? $\endgroup$ – Lewis Morris Feb 23 at 18:58
  • 1
    $\begingroup$ I am not suggesting, I am using it to explain a case. When you have less data, standard deviation is less stable. $\endgroup$ – Carlos Mougan Feb 23 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.