I am a student currently trying to create a classification model, however I am having difficulty understanding a weird overfitting problem.
A dataset of about 30 000 entries, 30 features. The data is sorted by date of entry. I split my data to 80% training & 20% testing. I get a training accuracy score of about 98% and a test accuracy of about 71% using random forest. When I remove 3 specific parameters, the overfitting disappears, and I get to 73% training accuracy and 68% test accuracy. Which means these 3 parameters cause a big deal of overfitting. The weird thing happens when I shuffle the data. With all the 30 parameters, the training accuracy remains 98% and the test accuracy gets up to 92%. Which for me indicates that these 3 features values change unexpectedly during the last month or so of the data (the data was sorted by date before shuffling) and shuffling them gives the model enough examples from this last month to pick up the sudden change. But plotting the mean of their values/day, for the whole spectrum of dates shows that they follow the same seasonality throughout the whole data, and there are no weird changes. Can someone please give me some ideas to explain why shuffling the data helps to reduce the overfit massively?