Time Series Machine Learning Feature Selection Problem

Question

I have to solve a time series model that can take one of two shapes. It can probably take more but here are the two I'm going to ask about. If you have other ideas they are of course welcome.

First Possible Model -

X(Dependent variable 'Spending') = X(lag1)...X(lagN) + X(Dummy variable when US) + X(Dummy Variable when Mexico) ... + Error term

Or Make a separate model for each Country like -

X(Total Spending in the US only) = X(lag1)...X(lagN) + Error term

X(Total Spending in the Mexico only) = X(lag1)...X(lagN) + Error term

… for Each country

Mathematically I can't decide what approach is better. I will use an F-Statistic, dickey fuller statistic to check the auto regression for stationarity and then compare the two models but I wanted to see what others thought of the theory and if you should ever include the dummy variables.

I'm looking more for an answer that includes mathematical reasoning.

Answer 1

I will give you some other idea of it, but without mathematical.

Like horaceT's mentioned, it's all about feature construction and feature selection, then you have many machine learning method which can do the regression.

So, just look it as regression model, not just the time series model. Then you will get the following model:

y(Dependent variable 'Spending') = X1(moving average term)...X2(lastquarter moving average term) + X3(country id) + X4(Fourier term) ... + Error term

The moving average can catch the dependent variable's trend, Fourier term can catch the dependent variable's cycle. Maybe you can construct other feature.

Finally, you can selection the features using tree-based model, such as xgboost.

Answer 2

In the first model, there are

1 (the constant) + N (the lagged variables) + C (dummy for countries ) - 1(because Dummy_US + Dummy_Mx + ... = 1) = N + C

coefficients to be fitted, here C = #countries.

In the second model, there are C*(N+1) coefficients to be fitted. You should use the second approach if the variance of the error terms are very different. If not, you can get the C*(N+1) parameters by fitting the model

X(Dependent variable 'Spending') = X(lag1) + ...+ X(lagN) + + X(lag1)*X(Dummy US) + ...+ X(lagN)*X(Dummy US) + X(Dummy US)
+ X(lag1)*X(Dummy Mx) + ...+ X(lagN)*X(Dummy Mx) + X(Dummy Mx) ... + Error term

(drop the dummy for one country, which will be the reference county).

This model involves C*(N+1) coefficients.

The coefficients for the reference country are:

 constant  
 coeff of X(lag1)
...
 coef of X(lagN)

The coefficients to forecast values for another country, for example the US, are:

 constant + coeff of X(Dummy US)
 coeff of X(lag1) + ceoff of X(lag1)*X(Dummy US)
...
 coef of X(lagN) + ceoff of X(lagN)*X(Dummy US)