2
$\begingroup$

One benefit of creating a linear model is that you can look at the coefficients the model learns and interpret them. For example, you can see which features have the most predictive power and which do not.

How, if at all, does feature interpretability change if we normalize (scale all features to 0-1) all our features vs. standardizing (subtract mean and divide by the standard deviation) them all before fitting the model.

I have read elsewhere that you 'lose feature interpretability if you normalize your features' but could not find an explanation as to why. If that is true, could you please explain?

Here are two screenshots of the coefficients for two multiple linear regression models I built. It uses Gapminder 2008 data and statistics about each country to predict its fertility rate.

In the first, I scaled features using StandardScaler. In the second, I used MinMaxScaler. The Region_ features are categorical and were one-hot encoded and not scaled.

Not only did the coefficients change based on different scaling, but their ordering (of importance?) did too! Why is this the case? What does it mean?

Linear Model coefficients using StandardScaler for preprocessing

Linear Model coefficients using MinMaxScaler for preprocessing

$\endgroup$
4
  • $\begingroup$ You seem to be interested in standardizing and min - max . Neither of two reflects normalizing included in your question ? $\endgroup$ – Subhash C. Davar Aug 27 '20 at 16:18
  • $\begingroup$ @SubhashC.Davar from Wikipedia: "normalization can have a range of meanings.[1] In the simplest cases, normalization of ratings means adjusting values measured on different scales to a notionally common scale" such as 0-1. Further down the page you will see under 'Examples' you will see 'Min-Max Feature Scaling' which 'brings all values in the range [0, 1]... and is also called unity-based normalization'. So, my question does reflect one kind of normalizing. $\endgroup$ – Adam Murphy Aug 28 '20 at 8:25
  • $\begingroup$ Min -Max indicates the range and in no case, it reflects 0 , 1. In statistics, we use 0,1 to reflect standard normal distribution. It has no connection with scaling technique and issues in measuring a phenomenon or a concept. $\endgroup$ – Subhash C. Davar Aug 28 '20 at 14:28
  • 1
    $\begingroup$ I understand :) I think you mean Max - Min to indicate the range, right? As this is the data science stack exchange, I think using normalization to mean "scale features to be in the range [0, 1]" is common practice. $\endgroup$ – Adam Murphy Aug 29 '20 at 10:35
3
$\begingroup$

When you have a linear regression (without any scaling, just plain numbers) and you have a model with one explanatory variable $x$ and coefficients $\beta_0=0$ and $\beta_1=1$, then you essentially have a (estimated) function:

$$y = 0 + 1x .$$

This tells you that when $x$ goes up (down) by one unit, $y$ goes up (down) by one unit. In this case it is just a linear function with slope 1.

Now when you scale $x$ (the plain numbers) like:

scale(c(1,2,3,4,5))
           [,1]
[1,] -1.2649111
[2,] -0.6324555
[3,]  0.0000000
[4,]  0.6324555
[5,]  1.2649111

you essentially have different units or a different scale (with mean=0, sd=1).

However, the way OLS works will be the same, it still tells you "if $x$ goes up (down) by one unit, $y$ will change by $\beta_1$ units. So in this case (given a different scale of $x$), $\beta_1$ will be different.

The interpretation here would be "if $x$ changes by one standard deviation...". This is very handy when you have several $x$ with different units. When you standardise all the different units, you make them comparable to some extent. I.e. the $\beta$ coefficients of your regression will be compareable in terms of how strong the variables impact on $y$ is. This is sometimes calles Beta-Coefficients or Standardised Coefficients.

A very similar thing happens when you normalise. In this case you will also change the scale of $x$, so the way how $x$ is measured.

Also see this handout.

$\endgroup$
1
  • 1
    $\begingroup$ This makes so much more sense now! Excellent explanation @Peter. If we normalize and all features are scaled from 0-1, the beta coefficient will still be interpreted as "if x increases by 1, y increases by beta". However, "increasing by 1" (after normalization) is the same as increasing by the entire range of values the feature takes. So are the coefficients telling us how y is impacted if we move from the lowest value of that feature (0) to the most extreme value (1)? $\endgroup$ – Adam Murphy Aug 25 '20 at 9:21
1
$\begingroup$

I believe with scaling, the coeff. are scaled by the same level i.e. Std. Deviation times with Standardization and (Max-Min) times with Normalization

If we look at all the features individually, we are basically shifting it and then scaling it down by a constant but $y$ is unchanged.

So, if we imaging a line in a 2-D space, we are keeping the $y$ same and squeeing the $x$ by a constant(Let's assume it = $C$).

This implies(Assuming Coeff. = Slope = $tan{\theta}$ = dy/dx) ,
the slope will also increase by the same amount i.e. $C$ times. (Since, dx has been divided by a constant($C$) but dy is same, so $tan{\theta}$ i.e. slope = $C$ * old_slope(i.e. the slope prior to scaling)

We can observe in this snippet that both the coef are in the ratio of the Standard deviation and (Max - Min) respectively w.r.t the unscaled coeff

import sys;import os;import pandas as pd, numpy as np
os.environ['KAGGLE_USERNAME'] = "10xAI" 
os.environ['KAGGLE_KEY'] = "<<Your Key>>" 

import kaggle
!kaggle datasets download -d camnugent/california-housing-prices

dataset = pd.read_csv("/content/california-housing-prices.zip")
y = dataset.pop('median_house_value')
x = dataset.iloc[:,:4]
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x,y)
old_coef = model.coef_  

x_s = (x-x.mean())/x.std()
model.fit(x_s,y)
std_coef = model.coef_  

print("###Ratio of Scaled Coeff and Std. Deviation times Standardized Coeff")
print(std_coef/(old_coef*x.std()))

x_n = (x-x.min())/(x.max()-x.min())
model.fit(x_n,y)
nor_coef = model.coef_  

print("###Ratio of Scaled Coeff and (Max - Min) times Normalized Coeff")
print(nor_coef/(old_coef*(x.max()-x.min())))

enter image description here

So, you can calculate the unscaled Coeff from Standardized and Normalized coeff.

On Importance

The order(Since it's sorted values) might change because the standard deviation will not be equal to (Max - Min).

But this should not impact the importance. Importance should be measured in original data space Or unit should be of standard deviation(as explained by Peter) Or (Max - Min) but that might be not very intuitive for every user.

$\endgroup$
3
  • $\begingroup$ I find some of the conclusions you draw hard to understand e.g. "but perpendicular is same, so... old_slope". Also, the code snippet you provided does not produce the output you wrote underneath. Could you please be more explicit, perhaps by adding print statements? $\endgroup$ – Adam Murphy Aug 25 '20 at 9:26
  • $\begingroup$ Updated a few parts and the output. Thanks!! $\endgroup$ – 10xAI Aug 28 '20 at 15:44
  • $\begingroup$ this is much better! Thank you for the update, your points are valuable :) Upvoted! $\endgroup$ – Adam Murphy Aug 29 '20 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.