I am developing a binary classification model using sklearn pipeline for preprocessing and a soft voting classifier (Adaboost and Extratrees with 50 estimators). The dataset (3 million rows) contains data from year 2020-2022 and is split into train/test/validation. The model is trained using the train dataset, tuned using the test dataset, and ultimately tested using the validation dataset. I also have some data from January 2023 as an additional holdout dataset (coming back to this).
When I run predictions on the validation dataset, this is what I get:
But when I run predictions on the January 2023 data, the performance is drastically worse, especially for the 0 class:
What could be the reason for this, and what measures can I take to reduce this effect?
Obviously, it seems like the model is overfitting, but I don't understand how the validation dataset performs so much better than the 2023 data seeing as both datasets are never touched during training. The dataset is resampled such that there are equally many data points for the 0 and 1 class, and I have tested several different model architectures resulting in similar behavior.
Further proof of overfitting is the following metrics on the train set:
And here is the feature importance for the two models (feature names removed due to confidentiality). Could the dominant features in the adaboost model mean that the models is overfitting based on those features?