K fold cross validation reduces accuracy

Question

I am working on a machine learning classifier and when I arrive at the moment of dividing my data into training set and test set Iwant to confron two different approches. In one approch I just split the dataset into training set and test set, while with the other approch I use k fold cross validation.

The strange thing is that with the cross validation the accuracy decreases, so if I have 0.87 with the first approch, with cross validation I have 0.86.

Shouldn't cross validation increase my accuracy? Thank's in advance.

Answer 1

Chance plays a big role when the data is split. For example maybe the training set contains a particular combination of features, maybe it doesn't; maybe the test set contains a large proportion of regular "easy" instances, maybe it doesn't. As a consequence the performance varies depending on the split.

Let's imagine that the performance of your classifier would vary between 0.80 and 0.90:

In one approch I just split the dataset into training set and test set

With this approach you throw the dice only once: maybe you're lucky and the performance will be close to 0.9, or you're not and it will be close to 0.8.

while with the other approch I use k fold cross validation.

With this approach you throw the dice k times, and the performance is the average across these $k$ runs. It's more accurate than the previous one, because by averaging over several runs the performance is more likely to be close to the mean, i.e. the most common case.

Conclusion: k-fold cross-validation isn't meant to increase performance, it's meant to provide a more accurate measure of the performance.

Answer 2

K-fold cross-validation trains k different models, each being tested on the observations not used in the learning procedure. There is no reason why you would get higher or lower scores in cross-validation, as you are not using the same model as in your reference case, neither the same test set. The approaches that you describe are different, although I would not recommend the cross-validation only.

Indeed, consider cross-validation as a way to validate your approach rather than test the classifier. Typically, the use of cross-validation would happen in the following situation: consider a large dataset; split it into train and test, and perform k-fold cross-validation on the train set only. The optimization of your model's hyperparameters is guided by the cross-validation score. Once you get the optimal hyperparameters setting, train your model with those parameters, on the full train set, and simply compute its accuracy on the test set.

Answer 3

A slight addition to Erwan's answer. It is important to be aware of the definition of "accuracy". It is not just the score, it is the quality of that score - which is what accuracy measures. By this I mean that the variability in your cross-validation score will be lower - and I think everyone would agree this is a higher quality estimate. In some cases as Erwan implies we may get a much higher score using the normal train-test split, but that's because we got lucky with the die roll. We could be equally unlucky too and get a lower score. You can be more confident with the CV score reported.

A sanity check using our intuition may also help. Ignore the superficial aim of getting a high score and consider the main underlying aim of generalising the underlying patterns in the data well (which score is just a measurement of) it makes intuitive sense to say that giving the algorithm would make achieving this goal easier. Which is in opposition to train-test split where we are actively restricting the algorithm from seeing some of the data.