4
$\begingroup$

I am a student currently trying to create a classification model, however I am having difficulty understanding a weird overfitting problem.

A dataset of about 30 000 entries, 30 features. The data is sorted by date of entry. I split my data to 80% training & 20% testing. I get a training accuracy score of about 98% and a test accuracy of about 71% using random forest. When I remove 3 specific parameters, the overfitting disappears, and I get to 73% training accuracy and 68% test accuracy. Which means these 3 parameters cause a big deal of overfitting. The weird thing happens when I shuffle the data. With all the 30 parameters, the training accuracy remains 98% and the test accuracy gets up to 92%. Which for me indicates that these 3 features values change unexpectedly during the last month or so of the data (the data was sorted by date before shuffling) and shuffling them gives the model enough examples from this last month to pick up the sudden change. But plotting the mean of their values/day, for the whole spectrum of dates shows that they follow the same seasonality throughout the whole data, and there are no weird changes. Can someone please give me some ideas to explain why shuffling the data helps to reduce the overfit massively?

$\endgroup$
1
$\begingroup$

There is a crucial assumption made by any supervised ML approach: both the training set and the set are samples drawn from the same population. This means that the model expects to see the same distribution of features in the test set as in the training set.

In order to make sure that this assumption is satisfied, it is important to shuffle the instances before splitting between training set and test set. This will avoid any visible or invisible bias due to the order in which the data was collected or assembled. That's the effect that you can observe in your first experiment: the main issue is not so much about overfitting due to the model relying too much on some parameters, it's that neither the training or test set are a random subsets of the population. It might make complete sense for the model to rely on these 3 parameters based on the training set, the problem is that they don't behave the same way in the test set. "behaving in the same way" includes their relation to other features, so the difference might not be visible by looking at these features independently. This bias (which probably affects more than the 3 features) causes a huge discrepancy when evaluating the model on the test set.

$\endgroup$
3
  • $\begingroup$ Thank you for this very clear explanation, but the model is supposed to predict the future, and therefore I believe it would be unrealistic to test it on old data the model has already seen in the past, hence the split without shuffling. Please correct me if I am wrong. $\endgroup$ – Aminek Jul 25 '19 at 18:06
  • $\begingroup$ Right, I meant to say something about this but I forgot: I assume that currently you are using a regular supervised learning algorithm, right? This kind of algorithm usually assumes that instances are independent from each other. Given that your data involves an evolution across time, it might make sense to use a method which takes into account the dependency between instances w.r.t time. Usually these methods work with "chunks of time" covering a sequence of data as instances, and take into account the dependency inside the sequence. $\endgroup$ – Erwan Jul 25 '19 at 23:33
  • $\begingroup$ Such methods are specialized in predicting what happens next in a sequence, so it should be more adapted to your case. I'm not very knowledgeable about these methods, but hopefully you can find what you need if you search for terms such as time series models for prediction or similar. $\endgroup$ – Erwan Jul 25 '19 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.