# How to properly use regression / tree based models for time-series data

Regression/tree-based models appear to treat each prediction as a memoryless process, namely given a feature vector $$\hat{x}_i$$, predict $$y_{i+1}$$, but previous states $$\hat{x}_{i-1}$$, $$\hat{x}_{i-2}, .. ,\hat{x}_0$$ aren't accounted for. This is a problem for non-Markovian processes.

One hack case is to attempt to mimic a Markovian process by adding features that are lagged versions of existing features, but more often than not this reduces the performance of the model, at least without serious feature engineering.

Even if all the features are made to be perfectly stationary, it's not clear that the state/feature vector $$\hat{x}_i$$ holds enough information to accurately generate predictions for $$y_{i+1}$$.

What are the best practices to overcome these limitations? Are there other ways to model time-series data using regression/tree-based models that I'm not aware of?

• In the general case (arbitrary data), it is not clear that the entire sequence of x since the beginning of time holds enough information to accurately generate predictions for y... More to the point - what is useful strategy depends on the particular data / process that generates the data Nov 7, 2022 at 19:38

I would argue that tree based models are not even Markovian per se, since those models are totally agnostic of time. Thus not even the formulation that from $$x_t$$ you predict $$\hat{y_{t+1}}$$ is misleading. Actually what you get using a tree is that from a vector $$x$$ you predict an output $$y$$ and this is it.
Regarding if it is not clear if the past contains enough information for the future this is an unsolved problem. Even when you use ARIMA like stuff, you don't test if the information is enough, but if the functional form of the model is of any use (like significance of fitted terms are statistically different from $$0$$).