I'm studying Data Scaling, and in particular the Standardization method. I've understood the math behind it, but it's not clear to me why it's important to give the features zero mean and unit variance.
Can you explain me ?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
The questions of whether and why it's important, depends on the context.
For gradient boosted decision trees, for example, it is not important - these ML algorithms "don't care" about monotone transformations to the data; they just look for points to split it.
For linear predictors, for example, scaling can improve interpretability of the results. If you'd like to think of the magnitude of the coefficients as some indication of how much a feature is affecting outcome, then the features has to be scaled somehow to the same area.
For some predictors, in particular NNs, scaling, and in particular scaling to a particular range, can be important for technical reasons. Some of the layers use functions that effectively change only within some area (similar to the hyperbolic-family of functions), and if the features are too much out of the range, saturation can occur. If this happens, numerical derivatives will work badly, and the algorithm might not be able to converge to a good point.
In case of zero mean, that is because some machine learning models do not include bias term in their representation so we have to move data around origin before feeding it to the algorithm to conpensate for lack of bias term. In case of unit variance, that is because lots of machine learning algorithms use some kind of distance (e.g. Euclidean) to decide or predict. If a particular feature has broad values (i.e. large variance), the distance will be highly affected by that feature and the effect of other features will be ignored. By the way, some optimization algorithms (including gradient descent) have better performance when data is standardized.
Hence, it is suggested to bring all features to same scale smaller enough to train easily. Below link also discusses similar concept. https://stats.stackexchange.com/questions/41704/how-and-why-do-normalization-and-feature-scaling-work