Time Series forecasting feature creation/engineering

Question

I'm new to time-series-forecasting and was wondering, whether in a single variable forecast e.g.: X -> Y the creation of additional features of X leads to an improvement when training. So if adding features like month, day-of-week, rolling mean or lagged series from X does improve Statistical models (Arima, Exponential Smoothing), Tree based models (XGB, Random Forest) or neural networks. These features would be fully correlated with X, so does it help, make predictions worse or have no effect? Thank you in advance

Answer 1

The answer is, it depends on your data.

Converting a univariate forecasting problem to a univariate with exogenous variables can help improve forecasts, particularly for GBDTs. For models that take exogenous variables performance is often (although not always) better with additional features. If features (X) help explain y, they will a model disentangle the patterns in the data. For example, here you see nbeatsx performing better than nbeats. You also mention date features, lags and MAs but you could also use cos and sin waves (fourier terms), these are extremely useful too.

While it is often preferable for exogenous variables to not be correlated with the dependents error terms, this is not a problem for neural networks or DTs which essentially reduce the influence of features that are not helpful in modelling. Worth noting correlated features may be more of an issue for random forest, and that it will affect interpretation even for GBDTs but not prediction. Its also important to point out that time series data is typically auto correlated so you have to draw this out in some way.

For ARIMA etc inherently create lags and moving averages, AR (auto regressive/lags) I (integrated/differencing), MA (moving average), this is also seen in seasonal components when given to a model (SARIMA). So you wouldn't create these directly, but other features may help a SARIMAX (x = exogenous). For example, if we predicted sales, price would help, as would say a promotional condition.

Now if you have noisy data, you may find the additional exogenous do not help, as I stated the tree based models and NNs typically won't be affected so much, so they will largely just create the same solution if the exogenous are included or not. Another solution in such a case is to aggregate your data to a lower temporal resolution.

Hope that helps.

Answer 2

Most tree-based models are not a good fit for time series forecasting. I'm not aware of any truly tree-based models that extrapolate outside of the training data as is required for forecasting.

As for the other models you mention, better input features for the particular application will result in better model performance. What makes a "better" features is extremely application dependent. I have used rolling moments with good success in a forecasting application.

Typically ARIMA and exponential smoothing models are univariate. In fact, every state space model (which exponential smoothing models fall under) has an ARIMA equivalent. The same can be said of their multivariate extensions (see Forecasting with Exponential Smoothing: the State Space Approach by Rob Hyndman). I bring up the fact they are univariate because your question seems like you want to feed multiple features to your model.

Features built from a time series do not need to be "fully correlated" with the time series as you say. As an example a transformation from time to something cyclic in time like day of the week or month will not be "fully correlated" with the original input.