Zero Mean and Unit Variance

Question

I'm studying Data Scaling, and in particular the Standardization method. I've understood the math behind it, but it's not clear to me why it's important to give the features zero mean and unit variance.

Can you explain me ?

Answer 1

The questions of whether and why it's important, depends on the context.

• For gradient boosted decision trees, for example, it is not important - these ML algorithms "don't care" about monotone transformations to the data; they just look for points to split it.

• For linear predictors, for example, scaling can improve interpretability of the results. If you'd like to think of the magnitude of the coefficients as some indication of how much a feature is affecting outcome, then the features has to be scaled somehow to the same area.

• For some predictors, in particular NNs, scaling, and in particular scaling to a particular range, can be important for technical reasons. Some of the layers use functions that effectively change only within some area (similar to the hyperbolic-family of functions), and if the features are too much out of the range, saturation can occur. If this happens, numerical derivatives will work badly, and the algorithm might not be able to converge to a good point.

Answer 2

In case of zero mean, that is because some machine learning models do not include bias term in their representation so we have to move data around origin before feeding it to the algorithm to conpensate for lack of bias term. In case of unit variance, that is because lots of machine learning algorithms use some kind of distance (e.g. Euclidean) to decide or predict. If a particular feature has broad values (i.e. large variance), the distance will be highly affected by that feature and the effect of other features will be ignored. By the way, some optimization algorithms (including gradient descent) have better performance when data is standardized.

Answer 3

• Whenever we start with any dataset in machine learning, we often assume that all the data features are equally important with respect to the output and one feature should not dominate over other feature. That’s GENERALLY the reason we choose to bring all the features to same scale.
  However, one may raise a doubt here that even if the features are not normalised then the weights assigned to it while learning may help the data set converge to expected output while training. The problem with this is that it will take really long to train and produce results.
• To choose specific number 0 as mean and variance 1 is just the ease to visualise and keeping such small numbers would help in faster training.

Hence, it is suggested to bring all features to same scale smaller enough to train easily. Below link also discusses similar concept. https://stats.stackexchange.com/questions/41704/how-and-why-do-normalization-and-feature-scaling-work