I have a dataset that has high collinearity among variables. When I created the linear regression model, I could not include more than five variables ( I eliminated the feature whenever VIF>5). But I need to have all the variables in the model and find their relative importance. Is there any way around it?. I was thinking about doing PCA and creating models on principal components. Does it help?.

  • $\begingroup$ Why can’t you include more than five variables? $\endgroup$
    – Dave
    Oct 31 at 2:30
  • $\begingroup$ Because VIF increases beyond 5 when I use more than 5 features. $\endgroup$
    – Naseef
    Oct 31 at 17:50
  • $\begingroup$ So VIF exceeds $5$…how does that impact your analysis? $\endgroup$
    – Dave
    Oct 31 at 20:24
  • $\begingroup$ Doesn't it mean high collinearity in the data? So that I can't keep those features $\endgroup$
    – Naseef
    Nov 1 at 1:17
  • $\begingroup$ But VIF of 4.5 also means that there is (multi)collinearity. How does VIF $>5$ impact your analysis? $\endgroup$
    – Dave
    Nov 1 at 3:05

PCA will generate „new“ (transformed) features which are orthogonal (non-correlated). However, since the original features are transformed, you can hardly claim to say a lot about the importance of (original) features based on PCA.

One obvious alternative would be to use a random forest (RF) to determine feature importance. Using tree based models (like RF or tree based boosting) you do not need to care about collinearity in the feature space.

  • $\begingroup$ But my principal components are still a linear combination of the original variables, right?. Can I distribute the feature importance of the principal components to the original variables somehow? $\endgroup$
    – Naseef
    Oct 26 at 5:49
  • $\begingroup$ Not sure about this. Tend to say it is not a good idea… $\endgroup$
    – Peter
    Oct 26 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.