What to do with large number of collinear variables?

Question

I have this time-series dataset that has 63 features, out of which 57 were manually engineered. While checking for collinearity, I get this correlation matrix: As can be seen there are a number of variables that are correlated/collinear. The ones that are deep red certainly need to be removed, but what about the ones on the bluer range? How do such variables (on the negative range of collinearity) effect regression models?

Also, I ran a recursive feature extraction process from sklearn.feature_extraction module and it has recommended me 39 features to be the best (at default settings). Is RFE the best strategy while dealing with such features?

Answer 1

First, if you're doing regression on a time series, you have to check for autocorrelation or your p-values and significance tests are going to be highly inaccurate. Also, the R^2 value you will get will be seriously misleading if you don't account for autocorrelation.

Secondly, if you are using lagged variables, it's highly likely that many of them are going to be correlated.

Collinearity affects linear regression models by making it difficult for the model to determine which coefficient is causing the effect on the dependent variable. The result is that between highly collinear variables, you will have inaccurate p-values and very small coefficients or even coefficients with the wrong signs.

Multicollinearity can be ignored when you are using some of those variables as control variables or when you don't care very much about interpretation; furthermore, collinearity doesn't matter when the variables of interest are not collinear.

Cross-validated has many good discussions on regression as well.