K-Fold Cross validation confusion?

Question

I am using K-Fold cross validation to test my trained model, but was amazed that for every K-fold the accuracy is different. For instance, if I use 5 K-fold, every fold has a different accuracy. So, which fold should I use? Is averaging all 5 of the folds the best choice? Secondly, why is the data-set split ratio (70/30) different in 5 fold Cross validation and 10 fold cross validation? Shouldn't it be the same?

Answer 1

The accuracy is different because there are k-classifiers made for each number of k-folds, and a new accuracy is found.

You don't select a fold yourself. K-Fold cross-validation is used to test the general accuracy of your model based on how you setup the parameters and hyper-parameters of your model fitting function.

What you do select is the number of folds, so in your example of 5 folds, it will do the following:

1. split up your training set into 5 different subsets (folds)
2. create a classifier for each of the 5 folds by using k-1 folds for fitting the model, and test the classifier accuracy using the fold left out

After it's done you can see how your classifier fared over the average of these folds.

If you're trying to find the optimal parameters to configure your model for the best accuracy you should be using grid search. Depending on your language the implementation will be different: python using sklearn.model_selection.GridSearchCV

, and R uses the 'carat' library and the train() function. Once you've run grid search you can either use the resulting model if you're programming in R or in python, you can add the new hyperparameters to your model fitting function and re-fit your model.

Your train/test ratio will depend on the number of folds. If you have 100 rows in your training set and you have 5 folds, then you'll have an 80/20 split train/test. If you have 10 folds it will be a 90/10 split. The cross-validation function for k-folds uses k-1 folds for fitting the model, and the fold left out is used for testing.