Binary classification performance difference between 0 and 1 class

Question

I have trained a binary Random Forest classifier on a dataset containing 7M rows. I also set aside a holdout validation set of 1M rows that the training pipeline never sees. The dataset consists of data from the last 6 years, and the holdout dataset contains data from the last year.

After splitting into X_train and X_test during training (random split), this is the classification report on X_test:

I.e. pretty good precision and recall on the 1 class, but not as good recall on the 0 class. However, when testing on the holdout dataset, I get the following performance:

I.e. notably worse performance than on X_test, especially on the 0 class which basically is as good as a coin flip.

I am expecting some decrease in performance metrics when testing on a holdout validation set of course, but not this much. Is this simply a case of overfitting? How should I go about remedying the difference in performance between the two classes? What does this kind of behavior tell about my data/model?

Answer 1

You have not addressed how many variables there are, what type are the variables.

How many are categorical?

How many levels are there in those categorical variables?

A very specific failure of decision trees is that with a lot of categorical variable level the tree over-trains on the categories and does not generalize well at all.

It is a common kind of instability that is not talked about openly often enough in my opinion.

Another thing you need to consider is that the pattern captured by the training data in the first 6-years has changed all or in part. It is an inherent possibility in any system over time.

A good place to start is to figure out if your data is appropriate to a decision tree in general and a random forest in particular based on the categorical variable presence.

Then make sure to do some tuning, explore the appropriate number of nodes and variables to work with and see if that improves consistency (your train my go down, but if it is more reflective of the test set it is a more robust model)

And finally you can work with some diagnostics, variable importance in a good place to start, see if you can figure out the most important variables and try again. It may help stabilize your model.

But before you do any of that, I would build charts and graphs of your data from the train and the test, do some tests to see if you have skew, kurtosis or heteroskedasticity. It may be there in the train or the test or both.

While random forests do not demand normally distributed data, if your train is to be predictive of the test, they should be similarly patterned. Especially over a dataset as large as 8-million rows.

Short of using unfit variables I think the temporal shift might be the culprit.