Does higher coefficient equate to high feature importance?

Question

I have red in this blog post that the higher the feature coefficient the more important the feature is.

However, the author did not mention why higher coefficient equates to higher importance, which made me confused.

Answer 1

It is not correct to claim that "higher" coefficients per se equate to more importance. The reason is that the scale of the data matters in linear models.

df = data.frame(y=c(1,1,1,0,0,0),x1=c(10,11,12,5,4,6), x2=c(11310,12520,10110,6010,5020,4010))

logit1 <- glm(y ~ x1 + x2, data = df, family = "binomial")
summary(logit1)


df$x2_alt = df$x2 / 1000
df$x2_alt

logit2 <- glm(y ~ x1 + x2_alt, data = df, family = "binomial")
summary(logit2)

Logit 1:

Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept) -7.274e+01  1.679e+05       0        1
x1           4.561e+00  1.072e+05       0        1
x2           4.450e-03  1.019e+02       0        1

Logit 2:

Coefficients:
              Estimate Std. Error z value Pr(>|z|)
(Intercept)    -72.743 167870.599       0        1
x1               4.561 107247.232       0        1
x2_alt           4.450 101948.329       0        1

As you can see, a linear transformation on $x_2$ changes the estimated coefficient in my dummy example above.

In order to compare coefficients in linear models, all $x$ need to be on the same scale (sometimes called beta regression). I.e. the $x$ need to have a standard deviation of one. If all $x$ are scaled, you can claim that a "higer" coefficient has a higher impact on $y$ (stronger change in "high" $x$ coefficient is associated with stronger change in $y$ compared to "low" $x$ coefficient).

However, this is not neccesarily the same thing as "feature importance", which means "high predictive power" of some $x$.

In the context of a Logit model you would use "Lasso" or "Ridge" (or Elastic Net) to "shrink" coefficients which are not very useful in making a prediction. See ISL, Ch. 6.2.2. In these models an additional "penalty term" is used to "shrink" coefficients which contribute little (or less than others). In the case of OLS this would look like:

sklearn.linear_model.LogisticRegression by default uses an l2 penalty (option penalty) which is "Ridge".

So overall, you can use Ridge/Lasso to select features (some see this as feature importance). However, the claim that "larger coefficients" are per se "more important" is not correct.