Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
I have been trying to understand how multicollinearity within the independent variables would affect the Linear regression model. Wikipedia page suggests that only when there is a "perfect" multicollinearity, one of the independent variables would have to be removed from training.
Now my question is that should we only remove one of the columns if the correlation is equal to +/- 1 or do we consider a threshold (say 0.90) after which it should be considered as perfect multicollinearity.
This depends on context. Computationally, only a correlation of +/- 1 is problematic, because then there is no unique solution to the OLS criterion. Very strong correlation between predictor variables will may inflate standard errors. This indicates that the parameter estimates become less precise with multicollinearity. Predictive accuracy is often not hurt much by this, but if you want to do inference (e.g., significance tests), it may be more of a problem. If predictors are very strongly correlated, you may then be better of by picking only the best predictors for your regression model, or doing some kind of dimension reduction first (e.g., PCA).
Often the Variance Inflation Factor (VIF) is used to determine if some variables are (too) strongly correlated. VIF = 10 is a well accepted threshold for incuding/excluding variables. So also variables with "too high" correlation should be excluded from linear regression ("OLS") as described here and here.
Multicolinearity is not necessarily a problem for regression, it is just a fact of life. Unless you can use a designed experiment, variables "in nature" are often correlated and you have to live with that. Of course, if nature was orthogonal (statistical) life would be simpler ...