How to properly use regression / tree based models for time-series data

Question

Regression/tree-based models appear to treat each prediction as a memoryless process, namely given a feature vector $\hat{x}_i$, predict $y_{i+1}$, but previous states $\hat{x}_{i-1}$, $\hat{x}_{i-2}, .. ,\hat{x}_0$ aren't accounted for. This is a problem for non-Markovian processes.

One hack case is to attempt to mimic a Markovian process by adding features that are lagged versions of existing features, but more often than not this reduces the performance of the model, at least without serious feature engineering.

Even if all the features are made to be perfectly stationary, it's not clear that the state/feature vector $\hat{x}_i$ holds enough information to accurately generate predictions for $y_{i+1}$.

What are the best practices to overcome these limitations? Are there other ways to model time-series data using regression/tree-based models that I'm not aware of?

Answer 1

I would argue that tree based models are not even Markovian per se, since those models are totally agnostic of time. Thus not even the formulation that from $x_t$ you predict $\hat{y_{t+1}}$ is misleading. Actually what you get using a tree is that from a vector $x$ you predict an output $y$ and this is it.

Since this estimator is not constraint functionally (does not require some specific form for hypothesis) it is flexible enough to incorporate any kind of features, which is good. The drawback comes from the same liberty, you need to have plenty of data and that data to be relevant to your problem. Incorporating lagged features increase exponentially the need of training data, and that comes from the same flexibility of the model. And of course, since there is no structure, feature engineering gains much more importance in the process.

If you would have used a model which has functional constrains, like ARIMA-like stuff you do not need exponentially more data since this is reduced by the assumed functional connection between variables. One the other hand, if the assumed structure does not hold for your problem, any amount of data is of no use.

Regarding if it is not clear if the past contains enough information for the future this is an unsolved problem. Even when you use ARIMA like stuff, you don't test if the information is enough, but if the functional form of the model is of any use (like significance of fitted terms are statistically different from $0$).

Since you limited your question to tree based model, I think you cannot do much more than be imaginative with your feature engineering (various combinations of features with products, transformations and so on) and adding more data are the only things I can imagine you can try.