1
$\begingroup$

I am trying to perform a multi linear regression model:

$$y_i = β_0 + β_1x_{i1} + β_2x_{i2} +... + β_px_{ip} + ε_i$$

where $$x_{i1}, x_{i2}, ..., x_{ip}$$ are highly correlated with each other (VIFs can be as low as 5 and high as 10).

I am just wondering if there exists a procedure with the following properties:

1) reduces the collinearity of the variables (e.g. VIFs should be lower than 5 after the procedure)

2) the variables after the procedure should maintain the original meanings/interpretations.. (so PCA and FA are out).

3) not dropping any of the variables. I should have all p original varaibles.. (So LASSO and RIDGE are out)

Improve this question
$\endgroup$
4
  • $\begingroup$ Lasso will remove completely, but Ridge will shrink and most likely keep all depending on the value of lambda that you pick. There is a relation between PCA and Ridge described that might be interesting to you: stats.stackexchange.com/questions/81395/… $\endgroup$
    – Steven M. Mortimer
    Aug 2, 2018 at 20:27
  • $\begingroup$ In addition, centering your variables will reduce multicollinearity $\endgroup$
    – Steven M. Mortimer
    Aug 2, 2018 at 20:28
  • $\begingroup$ @StevenM.Mortimer Thank you for your comment. Where can I learn more about centering? What are some potential problems there? $\endgroup$
    – JungleDiff
    Aug 2, 2018 at 20:29
  • $\begingroup$ @StevenM.Mortimer - what do mean by centering? Subtracting the mean of the dataset will not reduce multicollinearity. At best it will precent the VIF value from falsely accusing cases of multicollinearity. Given your three constraints, I am not sure it is possible to reduce the multicollinearity between variables. Any linear transformations will not do it, assuming you apply them to all variables. $\endgroup$
    – n1k31t4
    Aug 2, 2018 at 22:13

2 Answers 2

Reset to default
1
$\begingroup$

No - there is no standard analysis option to reduce collinearity without dimension reduction.

One possible option is to adjust the data instances. You can collect more data (variance inflation factor tends to go down as sample size goes up) or drop the specific instances that are primary drivers of the collinearity.

Improve this answer
$\endgroup$
1
$\begingroup$

What is the purpose of the model? Is it prediction or interpretation? Multi collinearity will effect the precision of the estimates and estimates can be unstable, so if purpose is interpretation than you cannot have multi collinearity. If the purpose is prediction you can model with multi collinearity it will not impact prediction. If collinearity is due to squared variables or interaction between variables i.e. structural, than centering the variables ( subtracting the variable by their mean) will remove some multi collinearity. If collinearity is due to data, then dropping variables removes multi collinearity.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.