1
$\begingroup$

I have a relatively small dataset consisting of 1432 samples.

I have trained a Random Forest Classifier and performed KFold CV. The results of running 10 Fold CV are as follows:

=== 10 Fold Cross Validation Scores ===

CVFold 1 = 90.2%
CVFold 2 = 87.6%
CVFold 3 = 86.7%
CVFold 4 = 86.7%
CVFold 5 = 83.9%
CVFold 6 = 75.8%
CVFold 7 = 87.2%
CVFold 8 = 82.8%
CVFold 9 = 86.1%
CVFold 10 = 89.3%

Mean Cross Validation Score:  85.6%

I am just not sure how to explain why there are such high variances between some of the folds, i.e from 75.8% in fold 6 to 90.2% in fold 1.

My understanding is that it is simply that the classifier found the samples in the 1st fold (90%) easier to classify than it did in fold 6 (75%) but I'm actually not entirely sure if that is the case.

I understand that each case is different but is it common to have such variances? And is it acceptable?

Edit: some details regarding my data

I have 5 classes, which are imbalanced:

class 1 - 5%

class 2 - 10%

class 3 - 15%

class 4 - 60%

class 5 - 10%

I am using SMOTE to balance the classes.

$\endgroup$
  • 2
    $\begingroup$ The cross validation score variance can be quite high when the dataset has few observations. What's the size of yours (number of samples)? $\endgroup$ – Romain Reboulleau Dec 2 '19 at 21:12
  • $\begingroup$ @RomainReboulleau I have a rather small dataset unfortunately with 1432 samples $\endgroup$ – codiearcher Dec 3 '19 at 9:59
  • $\begingroup$ Please note, accuraccy is by far worst existing metric, you can base anything on. Please use RepeatedKFold with at least 50 cycles of training and testing. It's hard to get some insight into metrics distribution with just 10 values. $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 3 '19 at 12:51
  • $\begingroup$ What is your class balance? $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 3 '19 at 12:51
  • 1
    $\begingroup$ You apply resampling only to train split. I suggest you use stratified cv to ensure you maintain your balance when splitting. $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 3 '19 at 13:27
1
$\begingroup$

It's indeed a quite large variation, but nothing alarming since 9 out of 10 folds are within the range 0.8 to 0.9.

There are many possible factors: yes a fold can be easier than another one by chance, but it also indicates that the training process is a bit unstable. Increasing the number of instances and/or reducing the number of features often helps reducing the variation.

| improve this answer | |
$\endgroup$
1
$\begingroup$

Honestly I would not balance the classes. I do not know how many entries you generate via SMOTE, but it looks to me like the classes are rather balanced, with the exception of the fourth one.

If you have enough data, I would simply use CrossValidation with stratified sampling for the classes, and you might even undersample slightly the bigger one.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.