Time series forecasting dilemma. Could feature engineering overcome time dependency?

Question

I keep reading articles about time series forecasting.

They all start from the same assumption: time series forecasting can't be treated as a regression/classification problem. It is time dependent, which means our target y at time t depends on what the value y was at t-1.

The motivation often comes with some example data showing some trend/seasonality. Also, other supporting points are:

1. The data distribution (mean, variance) changes over time.
2. Traditional train/test splits won't make sense since what is the point of forecasting January data with September data?

Fair enough. But let me try to point out this example. Let's say we have a simple timestamp, variable dataset and we are trying to predict the value at t+1

| timestamp | value |
| 01/01/2019| 10 |
| 01/02/2019| 12 |
...
| 31/12/2019| ??? |

What we know is that there is no trend it's very weekly-cyclic instead, which means the value at t will probably depend on its value at t-7 days. We also know that depending on whether it's a day during the week or during the weekend, data will change accordingly.

What prevents me to use some basic feature engineering and transform the example data as follow?

| timestamp | value_at_t_minus_7 | day_of_week | value |
| 01/01/2019| 11 | 02 | 10 |
| 01/02/2019| 12 | 03 | 12 |
...
| 31/12/2019| 10 | 02 | ??? |

It is not time dependant from a formal perspective but the correlation between its lagged values and the information about the day of the week should take me where I want to, being able to use now classic and flexible methods such as Random Forest, XGB and also splitting train and test (of course leaving the holdout set for validation) to get a good sense of how my model is performing.

Could anyone offer their input supporting with some proper motivation?

Thanks!

Answer 1

They all start from the same assumption: time series forecasting can't be treated as a regression/classification problem. It is time dependent, which means our target y at time t depends on what the value y was at t-1.

Time series forecasting must take into account time dependency, but it doesn't have to be the only source of information. Many complex forecasting tasks take into account external variables and time dependency altogether.

---

What prevents me to use some basic feature engineering and transform the example data as follow?

| timestamp | value_at_t_minus_7 | day_of_week | value |

Nothing prevents you from doing that. Moreover, that value_at_t_minus_7 is time dependency. day_of_week can also be thought as a simplification, or summarization of time dependency information - because studying seasonality we concluded that x can be explained with the value of one week ago. It's another way to make time explain the output.

When working with regressions (such as ARIMA or Tree-based models) you have to manually specify what is the shape of time dependency. Therefore the method you proposed is perfectly "legal" and actually very common. Good luck with your task.

Answer 2

You can use regression models such as RandomForestRegressor() to solve time-series problem at your own risk. Please remember, regression model will also capture trend and seasonality based on features such as day_of_year, month, and year. However, I be careful before generalizing use of regression models over time-series modes. Regression models may not completely fail on time-series data, but results from regression models may not be as reliable as a statistically correct time-series model.

e.g. suppose we have a time series data with a date(YYYYMMDD) column and a target column e.g. stock_value ( a series of numeric values corresponding to each time stamp). One can use following two approaches:

• Use time-series ML models such as ARIMA, LSTM; and forecast the future target values for a given date.
• Second, one can extract three features from time-stamp column namely day_of_year, month_of_year, and year. Now, you will have four columns including the target. Use 3 new features as independent variable and target as dependent variable. Next, do train_test_split, prepare the regression model ( linear regression, decision_tree_regression, random_forest, or xgboost. In this case, the prediction will be for a given date in future. The model will be able to predict the future values.

Remark: In one of my work, I had applied both regression and time-series model on a data with two column namely time-stamp and target. Upon comparing, I found RandomForestRegressormodel gave accuracy of 85.3% and time-series ARIMA model gave accuracy of 85.8% on the test data. Now, this is in no way a substantial gain by time-series model over regression model, nor should on conclude to generalize these findings.

I have personally compared accuracy of time-series model and regression model in one of the time-series data. However, I don't have enough evidence to generalize this to those data-sets which have noise; in such cases time-series models are better at removing noise before capturing trend and seasonality leading to their better performance.

Answer 3

You are indeed correct in using basic feature engineering to transform your data. Most of the approaches of using deep learning for time series forecasting do the same.

Suppose your time series data contains only one variable other than the timestamp and you are pretty sure with the assumption of your data having a periodicity of 1 week, you can transform your data to include all previous 7 days and let your machine learning model learn to give appropriate weights to each of the previous 7 days. Your columns will then look as follows -

timestamp | value_at_t_1 | value_at_t_2 | ... | value_at_t_7 | day_of_the_week | current_value

If your data contains more than one variable for each timestamp, you can create a dataset where each training sample is a sequence - a sequence of data points where each data point contains the original constituent variables. In your case, the sequence length will be 7, as the periodicity of your data is one week and each timestamp is a day.

Suppose your original dataset had shape=(M, N) where M is the number of timestamps and N is the number of variables corresponding to each timestamp then your transformed dataset will have shape=(num_samples, seq_len, N) where num_samples depends on your sampling technique (M/seq_len for non-overlapping samples, M-seq_len+1 for overlapping ones).

After this dataset is created you can train an LSTM-RNN which can predict the needed independent variable given data for past seq_len timesteps. Hope this helps!

Answer 4

Yes, you can use tabular regression algorithms for forecasting as long as you make sure you properly evaluate your forecasting algorithms (i.e. on future, not past data). The general approach of how to adapt regression algorithms to forecasting problems is described here.

You can also build hybrid machine learning models. For example, you may want to first de-seasonalize or detrend the data before applying your regression algorithm. Or you may want to fit a machine learning model on the residuals of a classical forecasting algorithm like ARIMA.

If you're interested, we're developing a toolbox that extends scikit-learn for exactly these use cases. Below is a simple example how to use sktime. We've also done some comparative benchmarking of these approaches on the M4 study.

import numpy as np
from sktime.datasets import load_airline
from sktime.forecasting.compose import ReducedRegressionForecaster
from sklearn.svm import SVR
from sktime.forecasting.model_selection import temporal_train_test_split
from sktime.performance_metrics.forecasting import smape_loss

y = load_airline()  # load 1-dimensional time series
y_train, y_test = temporal_train_test_split(y)  
fh = np.arange(1, len(y_test) + 1)  # forecasting horizon
regressor = SVR()  # any other scikit-learn regressor would work 
forecaster = ReducedRegressionForecaster(regressor, window_length=10)
forecaster.fit(y_train)
y_pred = forecaster.predict(fh)
print(smape_loss(y_test, y_pred))
>>> 0.139046791779424