Cross Validation - Why does more folds increase variation?

Question

Can someone explain why increasing the number of folds in a cross validation increases the variation (or the standard deviation) of the scores in each fold.

I've logged the data below. I'm working on the Titanic dataset and there is around 800 instances. I'm using a StratifiedKFold and accuracy scoring metric.

I thought that adding more data decreased variance - so if my understanding is correct, adding more folds would increase the amount of data supplied to each fit? But it appears that the more folds and LESS data passed in the lower the Standard Deviation (but the mean accuracy for each CV remains around the same)

{5: {'Mean': 0.8136965664427847, 'Std': 0.015594305964595902},
 15: {'Mean': 0.8239359698681732, 'Std': 0.0394725492730379},
 25: {'Mean': 0.823968253968254, 'Std': 0.07380525674642965},
 35: {'Mean': 0.8284835164835165, 'Std': 0.08302266965043076},
 45: {'Mean': 0.8207602339181288, 'Std': 0.09361950295425485},
 55: {'Mean': 0.8243315508021392, 'Std': 0.08561359961087428},
 65: {'Mean': 0.8273034657650041, 'Std': 0.10483277787806128},
 75: {'Mean': 0.8274747474747474, 'Std': 0.11745811393744522},
 85: {'Mean': 0.8240641711229945, 'Std': 0.12444299530668741},
 95: {'Mean': 0.8305263157894738, 'Std': 0.12484655607120225},
 105: {'Mean': 0.8243386243386243, 'Std': 0.1399822172135676},
 115: {'Mean': 0.8240683229813665, 'Std': 0.12916193497823075},
 125: {'Mean': 0.8249999999999998, 'Std': 0.13334396216138908},
 135: {'Mean': 0.8306878306878307, 'Std': 0.15391278842405914},
 145: {'Mean': 0.8272577996715927, 'Std': 0.1552827992878498},
 155: {'Mean': 0.8240860215053764, 'Std': 0.16756897617377703},
 165: {'Mean': 0.8270707070707071, 'Std': 0.16212344628562209},
 175: {'Mean': 0.824, 'Std': 0.16293498557341674},
 185: {'Mean': 0.8278378378378377, 'Std': 0.1664272446370702},
 195: {'Mean': 0.8284615384615385, 'Std': 0.17533175091718106},
 205: {'Mean': 0.8265853658536585, 'Std': 0.185808841661263},
 215: {'Mean': 0.8265116279069767, 'Std': 0.188431515175417},
 225: {'Mean': 0.8288888888888889, 'Std': 0.17685175489623095},
 235: {'Mean': 0.8294326241134752, 'Std': 0.19467536066874633},
 245: {'Mean': 0.8231292517006802, 'Std': 0.2009280149561644},
 255: {'Mean': 0.823202614379085, 'Std': 0.20790684270535614},
 265: {'Mean': 0.8254716981132075, 'Std': 0.2109826210610222},
 275: {'Mean': 0.8254545454545454, 'Std': 0.2144726806895627},
 285: {'Mean': 0.8242690058479532, 'Std': 0.2182928219064767},
 295: {'Mean': 0.823728813559322, 'Std': 0.22096355056065273}}

Answer 1

This has been a discussion for quite a while. For a more theoretical point of view you can find a good summary here.

From a practical point of view I'd look at it as follows. With increasing $k$ two things happen:

1. Your $k-1$ training folds increase in size
2. Your validation folds decreases in size

From the first point you can draw the conclusion that your $k$ models become more similar since your training data becomes more similar since you splitt off less data for the validation sets in the $k$-th fold. Which might lead to less between model-variance.

Moreover, since you are training on more data, the relative model complexity (i.e. compared to your data) decreases. In the bias-variance-trade-off graph (taken from [1])

this means we are moving towards the left, i.e. we trade less model variance for more model bias (please note that model variance here has a more general and conceptual meaning than the calculated variance or standard deviation between folds). The reason is that we are fitting a model with constant complexity to more data as $k$ increases, i.e. it becomes harder to learn the training data.

However, we are not only increasing the size of our training set with increasing $k$. We are also decreasing the validation set in size (see point two from above). Therefore, there might be higher between-fold variance with regards to our validation sets. This might be more relevant if the overall dataset carries higher variance or more outliers.

---

[1] "The Elements of Statistical Learning" by Hastie et al

Answer 2

As your test set in the Cross-validation becomes smaller the variation becomes bigger.

Get to the limit case when you do Leave One Out Cross Validation (LOOCV), there will be one instance in the test set. Some instances will have a good performance while others will be really bad. The variation will be high.