I am confused about how I choose the number of folds (in k-fold CV) when I apply cross validation to check the model. Is it dependent on data size or other parameters?
$\begingroup$
$\endgroup$
1
asked Feb 22, 2018 at 5:23
-
$\begingroup$ See also: Choice of k in k-fold cross-validation. $\endgroup$– AlanFeb 22, 2018 at 12:56
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
The number of folds is usually determined by the number of instances contained in your dataset. For example, if you have 10 instances in your data, 10-fold cross-validation wouldn't make sense. $k$-fold cross validation is used for two main purposes, to tune hyper parameters and to better evaluate the performance of a model.
In both of these cases selecting $k$ depends on the same thing. You must ensure that the training set and testing set are drawn from the same distribution. And that both sets contain sufficient variation such that the underlining distribution is represented. In a 10-fold cross validation with only 10 instances, there would only be 1 instance in the testing set. This instance does not properly represent the variation of the underlying distribution.
That being said, selecting $k$ is not an exact science because it's hard to estimate how well your fold represents your overall dataset. I usually use 5-fold cross validation. This means that 20% of the data is used for testing, this is usually pretty accurate. However, if your dataset size increases dramatically, like if you have over 100,000 instances, it can be seen that a 10-fold cross validation would lead in folds of 10,000 instances. This should be sufficient to reliably test your model.
In short, yes the number of folds depends on the data size. I usually stick with 4- or 5-fold. Make sure to shuffle your data, such that your folds do not contain inherent bias.
-
4$\begingroup$ What you describe in the second paragraph sounds like leave-one-out CV. I thought that wasn't that weird of a thing to use? $\endgroup$– JADFeb 22, 2018 at 8:01
-
$\begingroup$ Only time I think leave-one-out CV should be used is when the data is so limited that you do not have any other choice. This is because you will want to train on as much data as possible. However, you pay by introducing high variance. It's fine if needed, but best to avoid it. $\endgroup$– JahKnowsFeb 22, 2018 at 8:25
-
3$\begingroup$ @JahKnows On the contrary, leave-one-out CV gives the most training data (because you are only leaving one instance out of the training set). The only downside the the computational cost. $\endgroup$– TokkotFeb 22, 2018 at 8:43
-
$\begingroup$ @Tokkot that is precisely what I said. You will want to use LOO-CV with a small dataset in order to minimize your model's bias. More data in your training set, less bias. However, in doing so you will introduce variance because the testing set is not well representative of your distribution. This will affect the generalization of your model when subject to novel data. $\endgroup$– JahKnowsFeb 22, 2018 at 8:51
$\begingroup$
$\endgroup$
Depends on how much CPU juice you are willing to afford for the same. Having a lower K means less variance and thus, more bias, while having a higher K means more variance and thus, and lower bias.
Also, one should keep in mind the computational costs for the different values. High K means more folds, thus higher computational time and vice versa. So, one needs to find a sweet spot between those by doing a hyper tuning analysis.
Also, you need to keep the size of your data in mind. If your data is very less, then even using a k-fold crossval wouldn't make sense. So, you might want to use a leave-one-out CV (LOOCV).
