I have trained a binary Random Forest classifier on a dataset containing 7M rows. I also set aside a holdout validation set of 1M rows that the training pipeline never sees. The dataset consists of data from the last 6 years, and the holdout dataset contains data from the last year.

After splitting into X_train and X_test during training (random split), this is the classification report on X_test:

enter image description here

I.e. pretty good precision and recall on the 1 class, but not as good recall on the 0 class. However, when testing on the holdout dataset, I get the following performance:

enter image description here

I.e. notably worse performance than on X_test, especially on the 0 class which basically is as good as a coin flip.

I am expecting some decrease in performance metrics when testing on a holdout validation set of course, but not this much. Is this simply a case of overfitting? How should I go about remedying the difference in performance between the two classes? What does this kind of behavior tell about my data/model?


1 Answer 1


You have not addressed how many variables there are, what type are the variables.

How many are categorical?

How many levels are there in those categorical variables?

A very specific failure of decision trees is that with a lot of categorical variable level the tree over-trains on the categories and does not generalize well at all.

It is a common kind of instability that is not talked about openly often enough in my opinion.

Another thing you need to consider is that the pattern captured by the training data in the first 6-years has changed all or in part. It is an inherent possibility in any system over time.

A good place to start is to figure out if your data is appropriate to a decision tree in general and a random forest in particular based on the categorical variable presence.

Then make sure to do some tuning, explore the appropriate number of nodes and variables to work with and see if that improves consistency (your train my go down, but if it is more reflective of the test set it is a more robust model)

And finally you can work with some diagnostics, variable importance in a good place to start, see if you can figure out the most important variables and try again. It may help stabilize your model.

But before you do any of that, I would build charts and graphs of your data from the train and the test, do some tests to see if you have skew, kurtosis or heteroskedasticity. It may be there in the train or the test or both.

While random forests do not demand normally distributed data, if your train is to be predictive of the test, they should be similarly patterned. Especially over a dataset as large as 8-million rows.

Short of using unfit variables I think the temporal shift might be the culprit.

  • $\begingroup$ Thank you @bethanyP. To answer your questions, the dataset consists of 32 columns of which 17 are numerical and the rest are categorical. However, some of the categorical columns contain a large number of unique values which are all one-hot-encoded. The final dataset has 3.5M rows and X columns, which is split into a 80/20 train/test split. Your concern regarding decision trees and number of categorical variables might be a source for overfitting indeed. $\endgroup$
    – fendrbud
    Oct 4, 2022 at 7:24
  • $\begingroup$ On your questions related to the last year's data, I have no reason to believe that it has changed compared to the previous year, at least not to the degree that we see here, but this should be further validated with more rigorous data analysis. When it comes to model tuning, I have run several grid searches resulting in a ExtraTreesClassifier with n_estimators=100. $\endgroup$
    – fendrbud
    Oct 4, 2022 at 7:33
  • $\begingroup$ I will try to analyse the train and test data for the factors you mention, as well as pruning some of the categorical columns and/or grouping rare one-hot-encoded variables. $\endgroup$
    – fendrbud
    Oct 4, 2022 at 7:36
  • $\begingroup$ Typo in the first comment; The final one-hot-encoded dataset has 3.5M rows and 11k columns. $\endgroup$
    – fendrbud
    Oct 4, 2022 at 7:52
  • $\begingroup$ I am sure that the extreme number categorical variables is a part of the problem. If you use variable importance it could be useful. I assume based on your X_train, X_test you are working in python. One of the shorts is that it requires one-hot. If you know R and you can run this without one-hot encoding which allows you to see true variable importance (the how unexploded categorical) and made decisions about what to include. $\endgroup$ Oct 4, 2022 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.