fbprophet - adding regressor

Question

During linear regression classes in academia, it is taught that including trivial/irrelevant features to the model decreases its ability to predict more accurately.

In fbprophet, there is this function, add_regressor(), which allows us to add additional regressors to the model.

I want to know: - Is there a way to check whether added parameter/feature actually improves the model or it's actually trivial? - What should I look for during the process of adding regressors to fbprophet?

Of course I prefer a more intuitive, smart way rather than simply checking the evaluation metric with and without the added regressor, since the improvement can simply be noise and cause overfitting, decreasing the ability to generalize on unseen data.

Answer 1

No, there is nothing really better than recalibrating the model and see if it improve performance. Most techniques will be approximation of what you are doing with your main model. If you happen to find a simpler model that perfectly explain which variable plays a role in your main model and how, you should probably just use the simpler model.

Basic a priori techniques (looking at correlation with output, with other variables) will fail in the context of complex machine learning models.

One practical way to go nowadays is to look at feature importance. Those techniques depends on the model you use and will provide an importance value for each feature of each instance. If a feature has a low importance for all instances you should try removing it. (you can do that by batches of k least important features).

Answer 2

In linear regression models, underspecification of the model will cause the so called "omitted variable bias". In tendence it is better to have a larger model than a model with omitted variables.

Suppose your model looks like:

$$ y = X \beta + u, $$

while the true model (the data generating process, DGP) is:

$$ y = X \beta + Z \gamma + u. $$

In this case, when you solve

$$ E(\hat{\beta}) = (X'X)^{-1} X'y$$

and you substitute $y$ with the true model (true $y$), you end up with:

$$ E(\hat{\beta}) = (X'X)^{-1} X'(X \beta + Z \gamma + u),$$ $$ E(\hat{\beta}) = \beta + (X'X)^{-1} X'Z \gamma + E((X'X)^{-1} X'u),$$ $$ E(\hat{\beta}) = \beta + (X'X)^{-1} X'Z \gamma. $$

If the second term in the last equation is not zero, $\hat{\beta}$ is biased. The second term will be zero only if $X'Z=O$ (when two sets of regressors are mutually orthogonal, so no relevance of the regressor) or if $\gamma=0$ (in which case the model is not underspecified).

Standard options to check if a variable actually add to a model is to look at the adjusted $R^2$ and the Akaike or Schwarz Criteria. In case the adjusted $R^2$ goes up after adding an additional variable/feature, this is an indication that adding the variable is useful. Akaike/Schwarz are even better for nested models. You would choose the model with the lowest value for these criteria.