I have trained a binary Random Forest classifier on a dataset containing 7M rows. I also set aside a holdout validation set of 1M rows that the training pipeline never sees. The dataset consists of data from the last 6 years, and the holdout dataset contains data from the last year.
After splitting into
X_test during training (random split), this is the classification report on X_test:
I.e. pretty good precision and recall on the 1 class, but not as good recall on the 0 class. However, when testing on the holdout dataset, I get the following performance:
I.e. notably worse performance than on
X_test, especially on the 0 class which basically is as good as a coin flip.
I am expecting some decrease in performance metrics when testing on a holdout validation set of course, but not this much. Is this simply a case of overfitting? How should I go about remedying the difference in performance between the two classes? What does this kind of behavior tell about my data/model?