Linear Regression and scaling of data

Question

The following plot shows coefficients obtained with linear regression (with mpg as the target variable and all others as predictors).

For mtcars dataset (here and here) both with and without scaling the data:

How do I interpret these results? The variables hp and disp are significant only if the data is scaled. Are am and qsec equally important or is am more important than qsec? Which variable should one say are important determinants of mpg?

Thanks for your insight.

Answer 1

The fact that the coefficients of hp and disp are low when data is unscaled and high when data are scaled means that these variables help explaining the dependent variable but their magnitude is large, so the coefficients in the unscaled case have to be low.

In terms of "importance", I would say that the absolute value of the coefficients in the scaled case is a good measure of the importance, more than in the unscaled case, since there the magnitude of the variable is also relevant, and it should not.

Of course the more important variable is wt.

Answer 2

You can't really talk about significance in this case without standard errors; they scale with the variables and coefficients. Further, each coefficient is conditional on the other variables in the model, and collinearity actually seems to be inflating the importance of hp and disp.

Rescaling variables should not change the significance of results at all. Indeed, when I reran the regression (with the variables as is, and normalized by subtracting off the mean and dividing by the standard errors), each coefficient estimate (except the constant) had exactly the same t-stat as before scaling, and the F-test of overall significance remained exactly the same.

That is, even when all of the variables have been scaled to have a mean of zero and a variance of 1, there is no one size of standard error for each of the regression coefficients, so just looking at the magnitude of each coefficient in the standardized regression is still misleading about significance.

As David Masip explained, the apparent size of the coefficients has an inverse relationship with the magnitude of the data points. But even when the coefficients on disp and hp are huge, they are still not significantly different from zero.

In fact, hp and disp are highly correlated with each other, r=.79, so the standard errors on those coefficients are especially high relative to the coefficient magnitude because they are so collinear. In this regression, they're doing a weird counterbalancing, which is why one has a positive coefficient and one has a negative coefficient; it seems like a case of overfitting and does not seem meaningful.

A good way to see which variables explain the most variation in mpg is the (adjusted) R-squared. It is literally the percentage of the variation in y that is explained by the variation in the x variables. (Adjusted R-squared includes a slight penalty for each additional x variable in the equation, to counterbalance overfitting.)

A good way to see what's important--in light of the other variables--is to look at the change in adjusted R-squared when you leave out that variable from the regression. That change is the percentage of variance in the dependent variable that that factor explains, after holding constant the other variables. (Formally, you can test whether the left-out variables matter with an F-test; this is how stepwise regressions for variable selection work.)

To illustrate this, I ran single linear regressions for each of the variables separately, predicting mpg. The variable wt alone explains 75.3% of the variation in mpg, and no single variable explains more. However, many of other variables are correlated with wt and explain some of that same variation. (I used robust standard errors, which might lead to slight differences in standard error and significance calculations but will not affect coefficients or R-squared.)

+------+-----------+---------+----------+---------+----------+-------+
|      |   coeff   |   se    | constant |   se    | adj R-sq | R-sq  |
+------+-----------+---------+----------+---------+----------+-------+
| cyl  | -0.852*** | [0.110] |        0 | [0.094] |    0.717 | 0.726 |
| disp | -0.848*** | [0.105] |        0 | [0.095] |    0.709 | 0.718 |
| hp   | -0.776*** | [0.154] |        0 | [0.113] |    0.589 | 0.602 |
| drat |  0.681*** | [0.123] |        0 | [0.132] |    0.446 | 0.464 |
| wt   | -0.868*** | [0.106] |        0 | [0.089] |    0.745 | 0.753 |
| qsec |  0.419**  | [0.136] |        0 | [0.163] |    0.148 | 0.175 |
| vs   |  0.664*** | [0.142] |        0 | [0.134] |    0.422 | 0.441 |
| am   |  0.600*** | [0.158] |        0 | [0.144] |    0.338 | 0.360 |
| gear |  0.480*   | [0.178] |        0 | [0.158] |    0.205 | 0.231 |
| carb | -0.551**  | [0.168] |        0 | [0.150] |    0.280 | 0.304 |
+------+-----------+---------+----------+---------+----------+-------+

When all the variables are in there together, the R-squared is 0.869, and the adjusted R-squared is 0.807. So, throwing in 9 more variables to join wt just explains another 11% of the variation (or merely 5% more, if we correct for overfitting). (Many of the variables explained some of the same variation in mpg that wt does.) And in that full model, the only coefficient with a p-value under 20% is wt, at p= 0.089.