1
$\begingroup$

Assume the following scenario:

  1. I have four features: $x_1$, $x_2$, $x_3$, and $x_4$
  2. There are non-negligible multi-collinearity among the features.
  3. I want to predict $y$ (response variable) with those 4 features.
  4. I use simple multiple linear regression model: $y = a_1x_1 + a_2x_2 + a_3x_3 + a_4x_4$

Let's say that I want to understand the impact that different components of a chair have on the chair's retail price. For example:

$y\,\,\,$ = chair's retail price

$x_1$ = color of cushion used

$x_2$ = overall design of a chair

$x_3$ = strength of a chair

$x_4$ = softness of a chair

$x_1$ is completely independent, but other features are all somewhat impacted by the other features due to multicollinearity. For example, changing the color of cushion changes the design of a chair. Changing the design (structure) of a chair changes the strength of a chair.

I've heard that the analysis of regression coefficients are unreliable under severe multi-collinearity.

Assuming that the multiple regression model fits the chair price well, can I naively use each feature's regression coefficient to understand the impact of each feature on the response variable? If not, what technique should I use?

Ex 1: If I use a red cushion ($x_1$), I can increased the retail price by 3 dollars

Ex 2: If I use conference room style chair ($x_2$), I can increase the retail price by 12 dollars

$\endgroup$
  • $\begingroup$ how strong is the correlation between your $x$. $\endgroup$ – Peter Nov 10 '19 at 19:44
1
$\begingroup$

When you face multicollinearity, your regression coefficients will likely be biased, because under multicollinearity, the regression can not tell apart the different effects: https://datascience.stackexchange.com/a/57118/71442.

When you use only one variable at a time, you will face the omitted variable bias, because there are no other confounders, to which the regression could attribute relevant effects. https://en.wikipedia.org/wiki/Omitted-variable_bias

To my best knowledge, there is no easy-to-go way to mitigate one or both of the described effects. You should carefully check the correlation between your $x$ and decide if multicollinearity is a problem. If this is the case, and you believe (for theoretical reasons), that all the highly correlated $x$ are important, you could try to find other representations of (some) $x$ to mitigate multicollinearity, e.g. dummy/indicator representations.

I guess your chair example is generic, so I don‘t speculate on this. Maybe you can give some more background regarding the actual problem.

The discussion above refers to causal modeling. If you are only interested in making predictions (not statistical inference), you could look into shrinking coefficients using lasso.

$\endgroup$
  • $\begingroup$ I asked this question for self-study purpose, so I don't have an actual problem. Let's say that I have a customer who wants to quantify the impact of investing his money on feature 1 on his final profit y. There are many other features that predict the final profit y. Those features are all correlated to feature 1. Can you speculate on this? How would you quantify the impact of investing, say, $400, on feature 1 on the final profit, while it is obvious that the $400 investment will have synergy effect on the other features as well, which in chain will also impact the profit y? $\endgroup$ – Eric Kim Nov 10 '19 at 22:13
  • $\begingroup$ This would be a scenario in which you have a full control of only one feature, and the other features are difficult to control as you wish $\endgroup$ – Eric Kim Nov 10 '19 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.