0
$\begingroup$

In order to achieve scalable and robust time series forecast models, I am currently experimenting with metalearner ensembles. Note, that I am also using a global modeling approach, so all time series are "learning" from another. In my example I want to predict the monthly demand for 12 retail products one year ahead (4 years of training data available) I also choose different datasets to test the following.

As base models, I am fitting 6 XGBoost Models with 6 different learning rates from 0.001 to 0.65. The other parameters I do not tune (parsnip library defaults). For the metalearner I am also using an XGBoost Model with the following tuning grid:

enter image description here

Note, that I choose a very small range for the eta parameter!

The grid search resulted in mtry = 16, min_n = 13, tree_depth = 7 and a learning rate of 0.262. Trees were set to 1000 and early stopping parameter was set to 50 iterations.

However, my results are too good to be true (realistic) I guess. Below you can see the typical accuracy results from my predictions: enter image description here

As you can see, the predicted line of the ex post (training forecast on out-of-sample test set) almost perfectly matches with the actual values. Also there is an almost perfect bias, variance tradeoff. This seems odd, because it should be really hard to achieve in real world problems.

Now I am asking myself, if this approach is just exactly what I need, or if this

$\endgroup$
1

1 Answer 1

1
$\begingroup$

Without more details, it seems to me that you have a data leak problem.

How did you split the data in train/test? Notice that you're dealing with a time-series problem, so the standard random split wouldn't work. If you do a random split you would have a data leak, ie: your model would be trained with data from the future.

In time-series problems, you need to do a split based on time, for example, use all the data from the first 3 years to train the model and then evaluate the model with the remaining data of the last year.

$\endgroup$
1
  • $\begingroup$ In the end it was a coding error of mine. I did copy the wrong data object into the resampling tscv object. So indeed it was an unintentional data leak $\endgroup$
    – LGe
    Jun 9, 2022 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.