Understanding why shuffling reduces weirdly the overfit

Question

I am a student currently trying to create a classification model, however I am having difficulty understanding a weird overfitting problem.

A dataset of about 30 000 entries, 30 features. The data is sorted by date of entry. I split my data to 80% training & 20% testing. I get a training accuracy score of about 98% and a test accuracy of about 71% using random forest. When I remove 3 specific parameters, the overfitting disappears, and I get to 73% training accuracy and 68% test accuracy. Which means these 3 parameters cause a big deal of overfitting. The weird thing happens when I shuffle the data. With all the 30 parameters, the training accuracy remains 98% and the test accuracy gets up to 92%. Which for me indicates that these 3 features values change unexpectedly during the last month or so of the data (the data was sorted by date before shuffling) and shuffling them gives the model enough examples from this last month to pick up the sudden change. But plotting the mean of their values/day, for the whole spectrum of dates shows that they follow the same seasonality throughout the whole data, and there are no weird changes. Can someone please give me some ideas to explain why shuffling the data helps to reduce the overfit massively?

Answer 1

There is a crucial assumption made by any supervised ML approach: both the training set and the set are samples drawn from the same population. This means that the model expects to see the same distribution of features in the test set as in the training set.

In order to make sure that this assumption is satisfied, it is important to shuffle the instances before splitting between training set and test set. This will avoid any visible or invisible bias due to the order in which the data was collected or assembled. That's the effect that you can observe in your first experiment: the main issue is not so much about overfitting due to the model relying too much on some parameters, it's that neither the training or test set are a random subsets of the population. It might make complete sense for the model to rely on these 3 parameters based on the training set, the problem is that they don't behave the same way in the test set. "behaving in the same way" includes their relation to other features, so the difference might not be visible by looking at these features independently. This bias (which probably affects more than the 3 features) causes a huge discrepancy when evaluating the model on the test set.