1
$\begingroup$

I am developing a binary classification model using sklearn pipeline for preprocessing and a soft voting classifier (Adaboost and Extratrees with 50 estimators). The dataset (3 million rows) contains data from year 2020-2022 and is split into train/test/validation. The model is trained using the train dataset, tuned using the test dataset, and ultimately tested using the validation dataset. I also have some data from January 2023 as an additional holdout dataset (coming back to this).

When I run predictions on the validation dataset, this is what I get:

validation dataset

But when I run predictions on the January 2023 data, the performance is drastically worse, especially for the 0 class:

2023 data

What could be the reason for this, and what measures can I take to reduce this effect?

Obviously, it seems like the model is overfitting, but I don't understand how the validation dataset performs so much better than the 2023 data seeing as both datasets are never touched during training. The dataset is resampled such that there are equally many data points for the 0 and 1 class, and I have tested several different model architectures resulting in similar behavior.

Further proof of overfitting is the following metrics on the train set:

train data

And here is the feature importance for the two models (feature names removed due to confidentiality). Could the dominant features in the adaboost model mean that the models is overfitting based on those features?

feature importances

$\endgroup$

1 Answer 1

3
$\begingroup$

I am not sure why you decided to resample your training and test data. The data in your validation data set is fairly balanced and you don't need to increase your sample size given the size of your dataset. So it looks to me this is not necessary at all.

One problem that could have occurred here is that your train/test/validation split did not take to into account the temporal ordering of the data. You might want to choose a validation set that uses e.g. all data from October-December 2022. Ignoring the temporal nature of the data might led to data leakage, i.e. the model obtained wrongly data from the future to predict the past. This could explain your great performance in training and on the validation set.

Less likely but still possible is model drift. The data in 2023 reflects a change in the real world that influences the nature of your data and the model has not yet learned the changing patterns.

$\endgroup$
2
  • $\begingroup$ Thank you. I tested your comment regarding temporal ordering by splitting the dataset like you mentioned, and indeed this resulted in worse performance on the validation set (similar to the performance when predicting on the 2023 data in the original post). I still get an f1=99% on the train set, but a f1=67% on the test set now. $\endgroup$
    – fendrbud
    Commented Feb 27, 2023 at 9:46
  • $\begingroup$ Okay great. Now you know in fact that you do overfit on the training data. Maybe you want to try regularization in order to punish over-complex models/decision trees. $\endgroup$
    – danielOh
    Commented Feb 27, 2023 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.