Although I can understand the significance of feature scaling in some cases (e.g. when gradient descent is involved), I don't feel I understand the necessity of this process in general. But there a lot of people that suggest feature scaling prior to the training phase.

Take for example, methods that use kernels such as the RBF.

$$\mathrm{RBF} = \exp{\left(-\frac{\rVert\mathbf{x}-\mathbf{x}'\lVert^2}{\gamma}\right)}$$

Two examples are: the t-SNE algorithm (for visualization purposes) and Gaussian Processes (for regression). Lets say that two features $x_1$ and $x_2$ are in the ranges $[0, 10]$ and $[0,1000]$, respectively. What justifies the normalization step prior to training phase (or fit-transform phase in case of t-SNE)?

Common arguments take the following form:

When distance metrics are involved, feature scaling is needed so that all features get equal weight.

If we have three points, $$\begin{align} a=(5, 200) \quad &| \quad a'=(0.5, 0.20)\\ b=(9, 220) \quad &| \quad b'=(0.9, 0.22)\\ c=(1, 900) \quad &| \quad c'=(0.1, 0.90) \end{align}$$

then for the distances between the $3$ points ordering is not conserved, that is:

$$ab < bc < ac \quad | \quad a'b' < a'c' < b'c' $$

This example is consistent with the quoted statement since in the non-normalised case the $\Delta x_2$ differences dominate in the distance calculation.

Is there any other reason that justifies feature scaling? Additionally, are there any cases that normalization can decrease the performance?


1 Answer 1


I found normal to be confusing and my belief is that often times there is no proper solution. I might be wrong, but I will share my thoughts on that.

In order to understand the complexity of the issue we should consider all parties involved or potentially affected by that: the model, the variable relationships from data, the samples involved and how they were collected and the domain knowledge.

Not all models are equal. Models can differ in multiple ways. Most models assumes something about the regression function which affects the types of hypotheses they can express. Linear models produces hypotheses in the form of linear combinations, for example, others not. Even if the produced hypotheses are identical, the way they arrive at a result might be different. Scaling can affect that in multiple ways. For a linear regression with intercept a standard scaling (subtract mean, divide by sd) will be absorbed into coefficients, if there is an intercept, while a min max scaling might be a disaster since might break further assumptions like heteroskedastic variance. The opposite is also possible since many times we scale variables to meet some assumptions required by the model. This is the case with Box-Cox transformation and others, used often to make the variance heteroskedastic. The method used to fit the model also play a factor. There are models which needs scaling in small number intervals in order to work, like sigmoid gradients in backpropagation. Other times it does have no effect, like monotone transformation on random forests.

When we use data to fit a regression function on independent variable we hope that variables are correlated in some way with the target. But connections exists between variables also. Those connections can be destroyed by scaling or can be improved. For example having longitude and latitude, if scaled separately and the sample is not representative we can destroy a lot. On the other hand we might have two plane coordinate variables, one measured in inch and other measured in centimeters, for whatever reason. A scaling into a similar range might help.

There are many cases in which a scaling can help or makes the problem worse. The question is how can we know if we have to apply or not some transformations? If we have strong domain expertise and if we have enough information about the collected measurements we might improve the situation. Knowledge about measurement units, measurement errors, relations between variables and connections (rarely clear like physics laws, but some connections could be made, like human heights and weights, and so on) can help to make decisions. But the truths is about the usefulness of those transformations is impossible to assess.

My personal approach when I am considering transformations is to find a reason for any kind of transformation. That reason, as a said, can be motivated by model hypothesis, by fitting, by domain knowledge if any, or anything connected with the process. If I find no reason to do feature scaling, I simply do not. Even if I think there is a strong reason, it remains to understand and justify what kind of scaling, which should be the parameters, and ultimately, I am ready to admit that it might be not useful. In general I think applying blindly feature scaling, like any kind of transformation is a really bad habit. I prefer to struggle to understand, before any change.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.