Why do I get an almost perfect fit as well as bias variance tradeoff with my time series forecast?

Question

In order to achieve scalable and robust time series forecast models, I am currently experimenting with metalearner ensembles. Note, that I am also using a global modeling approach, so all time series are "learning" from another. In my example I want to predict the monthly demand for 12 retail products one year ahead (4 years of training data available) I also choose different datasets to test the following.

As base models, I am fitting 6 XGBoost Models with 6 different learning rates from 0.001 to 0.65. The other parameters I do not tune (parsnip library defaults). For the metalearner I am also using an XGBoost Model with the following tuning grid:

Note, that I choose a very small range for the eta parameter!

The grid search resulted in mtry = 16, min_n = 13, tree_depth = 7 and a learning rate of 0.262. Trees were set to 1000 and early stopping parameter was set to 50 iterations.

However, my results are too good to be true (realistic) I guess. Below you can see the typical accuracy results from my predictions:

As you can see, the predicted line of the ex post (training forecast on out-of-sample test set) almost perfectly matches with the actual values. Also there is an almost perfect bias, variance tradeoff. This seems odd, because it should be really hard to achieve in real world problems.

Now I am asking myself, if this approach is just exactly what I need, or if this

Answer 1

Without more details, it seems to me that you have a data leak problem.

How did you split the data in train/test? Notice that you're dealing with a time-series problem, so the standard random split wouldn't work. If you do a random split you would have a data leak, ie: your model would be trained with data from the future.

In time-series problems, you need to do a split based on time, for example, use all the data from the first 3 years to train the model and then evaluate the model with the remaining data of the last year.