1
$\begingroup$

I have built a correlation matrix to check multicollinearity in a regression model. But how to interpret this?

enter image description here

  1. Can we say that there is a certain correlation value from which the independent variables related to this correlation should be removed from the model? Is this the value over 0.5?
  2. Are there any better solutions than removing the conflicting variables (when correlation value is too big)? I tried normalization but it didn't help.
$\endgroup$
2
  • $\begingroup$ You can try to apply PCA or TruncatedSVD to get/transform and select only the best and meaningfull features. Or maybee apply RFC and look for feature_importances_ attribute to help you decide which features have to be kept. $\endgroup$
    – Malo
    Commented Jul 7, 2021 at 22:19
  • $\begingroup$ Despite how much sense it makes at first, dropping variables has its problems. $\endgroup$
    – Dave
    Commented Mar 8, 2022 at 10:52

2 Answers 2

1
$\begingroup$

Yes; selecting based on the correlation coefficient, which I'll call $r$, is a valid option. It doesn't necessarily have to be $|r|>0.5$, but keep in mind that the lower you go, the more likely you are to lose valuable information. You may also decide that you wish to eliminate a certain number of features, $k$, and choose these based on the $k$-highest correlation coefficients.

If the reason why you want to eliminate variables is because you're worried about redundancy between features harming your predictivity, I would consider eliminating pH and stopping there, since it correlates with so many other variables. If you simply don't want to deal with too many variables, perhaps start eliminating the ones that correlate with pH (but not pH). I would prioritize elimination based on what makes sense in the real world, especially if you do not have a lot of data samples (meaning some of those high $|r|$ could be influenced by small sample size). E.g., I'm guessing you weren't surprised by pH being correlated with fixed acidity, since pH is an indication of how acidic or basic a compound is.

As for alternative solutions have you considered ridge regression? This is a supervised approached (considers labels/dependent variable), so it may not be appropriate for your problem. It's nice because it doesn't require as much decision making, although you may not be happy with the number of variables it sets to zero (eliminating them). Sometimes it eliminates too few, or maybe even too many, depending on your data and modeling needs. If you don't need to keep the original variables intact, PCA is a very popular unsupervised option for reducing features/redundancy between features while trying to preserve explained variance in the original data. This, theoretically, means you're less likely to lose predictivity than elimination through unsupervised approaches like the one you've proposed.

$\endgroup$
0
$\begingroup$

Regarding 1) Usually you would check this based on "tests" like the VIF (see this post for details). I personally would not be too worried with a correlation of 0.5.

Regarding 2) Not really since you would need to "decorrelate" the variables and you can only do linear transformations (when you want to preserve the variable(s)). What you can do is to make projections, e.g. by principle components, but this will totally change the variable set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.