Why do we convert skewed data into a normal distribution

Question

I was going through a solution of the Housing prices competition on Kaggle (Human Analog's Kernel on House Prices: Advance Regression Techniques) and came across this part:

# Transform the skewed numeric features by taking log(feature + 1).
# This will make the features more normal.
from scipy.stats import skew

skewed = train_df_munged[numeric_features].apply(lambda x: skew(x.dropna().astype(float)))
skewed = skewed[skewed > 0.75]
skewed = skewed.index

train_df_munged[skewed] = np.log1p(train_df_munged[skewed])
test_df_munged[skewed] = np.log1p(test_df_munged[skewed])

I am not sure of what is the need for converting a skewed distribution into a normal distribution. Please, can someone explain in detail:

1. Why is this being done here? or How is this helpful?
2. How is this different from feature-scaling?
3. Is this a necessary step for feature-engineering? What is likely to happen if I skip this step?

Answer 1

You might want to interpret your coefficients. That is, to be able to say things like "if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$".

For your coefficients to be interpretable, linear regression assumes a bunch of things.

One of these things is no multicollinearity. That is, your $X$ variables should not be correlated against each other.

Another is Homoscedasticity. The errors your model commits should have the same variance, i.e. you should ensure the linear regression does not make small errors for low values of $X$ and big errors for higher values of $X$. In other words, the difference between what you predict $\hat Y$ and the true values $Y$ should be constant. You can ensure that by making sure that $Y$ follows a Gaussian distribution. (The proof is highly mathematical.)

Depending on your data, you may be able to make it Gaussian. Typical transformations are taking the inverse, the logarithm or square roots. Many others exist of course, it all depends on your data. You have to look at your data, and then do a histogram or run a normality test, such as the Shapiro-Wilk test.

These are all techniques to build an unbiased estimator. I don't think it has anything to do with convergence as others have said (sometimes you may also want to normalize your data, but that is a different topic).

Following the linear regression assumptions is important if you want to either interpret the coefficients or if you want to use statistical tests in your model. Otherwise, forget about it.

Applying the logarithm or normalizing your data, is also important because linear regression optimization algorithms typically minimize $\|\hat y - y\|^2$, so if you have some big $y$ outliers, your estimator is going to be VERY concerned about minimizing those, since it is concerned about the squared error, not absolute error. Normalizing your data is important in those case and this is why scikit-learn has a normalize option in the LinearRegression constructor.

Answer 2

The skewed data here is being normalised by adding one(one added so that the zeros are being transformed to one as log of 0 is not defined) and taking natural log. The data can be nearly normalised using the transformation techniques like taking square root or reciprocal or logarithm. Now, why it is required. Actually many of the algorithms in data assume that the data science is normal and calculate various stats assuming this. So the more the data is close to normal the more it fits the assumption.

Answer 3

Because data science is just statistics at the end of the day, and one of the key assumptions of statistics is the Central Limit Theorem. So this step is being done because some subsequent step uses stats techniques that rely on it.