3
$\begingroup$

I have been trying to understand how multicollinearity within the independent variables would affect the Linear regression model. Wikipedia page suggests that only when there is a "perfect" multicollinearity, one of the independent variables would have to be removed from training.

Now my question is that should we only remove one of the columns if the correlation is equal to +/- 1 or do we consider a threshold (say 0.90) after which it should be considered as perfect multicollinearity.

Improve this question
$\endgroup$

4 Answers 4

Reset to default
6
$\begingroup$

This depends on context. Computationally, only a correlation of +/- 1 is problematic, because then there is no unique solution to the OLS criterion. Very strong correlation between predictor variables will may inflate standard errors. This indicates that the parameter estimates become less precise with multicollinearity. Predictive accuracy is often not hurt much by this, but if you want to do inference (e.g., significance tests), it may be more of a problem. If predictors are very strongly correlated, you may then be better of by picking only the best predictors for your regression model, or doing some kind of dimension reduction first (e.g., PCA).

Improve this answer
$\endgroup$
3
$\begingroup$

Often the Variance Inflation Factor (VIF) is used to determine if some variables are (too) strongly correlated. VIF = 10 is a well accepted threshold for incuding/excluding variables. So also variables with "too high" correlation should be excluded from linear regression ("OLS") as described here and here.

Improve this answer
$\endgroup$
3
$\begingroup$

Multicolinearity is not necessarily a problem for regression, it is just a fact of life. Unless you can use a designed experiment, variables "in nature" are often correlated and you have to live with that. Of course, if nature was orthogonal (statistical) life would be simpler ...

All of this is much discussed at our sister site Cross Validated, see for instance Is multicollinearity really a problem?

An economics' blog making fun of unnecessary preoccupation with colinearity is at Multicollinearity and Micronumerosity 14.

Improve this answer
$\endgroup$
0
$\begingroup$

The following paper explains very well the trade-off in removing variables.

See (2005). Graphical Views of Suppression and Multicollinearity in Multiple Linear Regression. The American Statistician: Vol. 59, No. 2, pp. 127-136.

Addendum: the paper studies the balancing act between colinearity effects and model fit, i.e., whether suppression and enhancement effects in regression offset colinearity issues.

Improve this answer
$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Oct 28, 2021 at 2:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.