2
$\begingroup$

I am using K-Fold cross validation to test my trained model, but was amazed that for every K-fold the accuracy is different. For instance, if I use 5 K-fold, every fold has a different accuracy. So, which fold should I use? Is averaging all 5 of the folds the best choice? Secondly, why is the data-set split ratio (70/30) different in 5 fold Cross validation and 10 fold cross validation? Shouldn't it be the same?

$\endgroup$
  • $\begingroup$ Cross-validation is for picking a model that gives better accuracy (or any metric) on various partitions you had made on your data. It is not for the selection of fold, it is for the selection of model. Generally avg. (accuracy) over the all test sets is taken for selection. And finally in 5-fold validation, you divide your data into five sets and you can use your choice number of folds for training, testing and validation. (60:40 or 80:20, 60:20:20). $\endgroup$ – Kiritee Gak May 23 '17 at 10:44
  • $\begingroup$ what should be the ideal folds for 70:15:15 $\endgroup$ – Case Msee May 23 '17 at 10:46
4
$\begingroup$

The accuracy is different because there are k-classifiers made for each number of k-folds, and a new accuracy is found.

You don't select a fold yourself. K-Fold cross-validation is used to test the general accuracy of your model based on how you setup the parameters and hyper-parameters of your model fitting function.

What you do select is the number of folds, so in your example of 5 folds, it will do the following:

  1. split up your training set into 5 different subsets (folds)
  2. create a classifier for each of the 5 folds by using k-1 folds for fitting the model, and test the classifier accuracy using the fold left out

After it's done you can see how your classifier fared over the average of these folds.

If you're trying to find the optimal parameters to configure your model for the best accuracy you should be using grid search. Depending on your language the implementation will be different: python using sklearn.model_selection.GridSearchCV

, and R uses the 'carat' library and the train() function. Once you've run grid search you can either use the resulting model if you're programming in R or in python, you can add the new hyperparameters to your model fitting function and re-fit your model.

Your train/test ratio will depend on the number of folds. If you have 100 rows in your training set and you have 5 folds, then you'll have an 80/20 split train/test. If you have 10 folds it will be a 90/10 split. The cross-validation function for k-folds uses k-1 folds for fitting the model, and the fold left out is used for testing.

$\endgroup$
  • 1
    $\begingroup$ perfect answer.What i am doing is,using k=4,my code runs for 4 times in a loop and for every k there is different classification accuracy as you also mentioned.At the end,I average the results of these 4 classification results.Is it right approach?.Secondly,is there any matlab function to configure optimal parameters? $\endgroup$ – Case Msee May 26 '17 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.