XGBoost gives same answers for parameters not in training range

Question

I have a data set (CPP(Monthly average), Channel, Show, Month, Year) for marketing return prediction. Data set has ads from 2022 to 2023 with about 130k points. I have a model, XGBOOST, and Train, test sets works good but my main goal is to get a guess for january 2024 which I know not the ideal use case for ML. But I can have some decent error margin and it would still be usefull My question is when I give this model january 2024 it makes the same guesses as january 2023 and this repeating goes on as the year does, but not january 2022 or feburary 2022. My guess is that this happens because of the tree structure of this model. 2024 goes through the same way as 2023. How can I solve this, different models don't perform well as the dataset is quite random in reality but xgboost seemed to do fine

what else can you suggest for a project like this neural networks with forecasting methods maybe?

Answer 1

The problem is that your dataset has little information to work with (only 1 continuous and 4 discrete variables).

The trees resulting from this will easily overfit on the data because there are not as many possible configurations. Thus resulting in identical predictions.

Answer 2

Your guess...

... is probably correct.

The year 2024 was not present in the training data, so for this input, XGBoost has to take a split based on previously learned data. Since xgb splits numbers by a threshold and 2024 ist larger than the largest trained value (2023), it will always take the same split as 2023.

What can you do?

There are ways to use xgb for extrapolation tasks, such as yours. Mainly, you can transform or enrich your data in a way that the difference between 2024 and 2023 can be learned. This can be done by:

Different target definition

Instead of predicting the values for a month, you could predict differences to a previous month. The question (that you will have to answer) is how much look-ahead to you want. Do you want to predict the difference between Dec. 2023 and Jan. 2024 or does it make mor sense to predict the difference between Jan. 2023 and Jan. 2024?

Feature Engineering

You can enrich your features, so that sample will differ between 2023 and 2024, even if all of your base features (except the year) are the same. You could for example introduce some statistics about the previous months, like the average, variance, ... over some feature. If you look at one feature and 3 month, you would get 3 new features. If you use mean and std for one original feature and 3 month, you already have 6 new features.

Doing so will easily get you new features from which xgb can learn the difference between 2023 and 2024. Now you need to understand the use case and the data and figure out what makes sense, here.

You can even combine both approaches.