# What justifies feature scaling?

Although I can understand the significance of feature scaling in some cases (e.g. when gradient descent is involved), I don't feel I understand the necessity of this process in general. But there a lot of people that suggest feature scaling prior to the training phase.

Take for example, methods that use kernels such as the RBF.

$$\mathrm{RBF} = \exp{\left(-\frac{\rVert\mathbf{x}-\mathbf{x}'\lVert^2}{\gamma}\right)}$$

Two examples are: the t-SNE algorithm (for visualization purposes) and Gaussian Processes (for regression). Lets say that two features $$x_1$$ and $$x_2$$ are in the ranges $$[0, 10]$$ and $$[0,1000]$$, respectively. What justifies the normalization step prior to training phase (or fit-transform phase in case of t-SNE)?

Common arguments take the following form:

When distance metrics are involved, feature scaling is needed so that all features get equal weight.

If we have three points, \begin{align} a=(5, 200) \quad &| \quad a'=(0.5, 0.20)\\ b=(9, 220) \quad &| \quad b'=(0.9, 0.22)\\ c=(1, 900) \quad &| \quad c'=(0.1, 0.90) \end{align}

then for the distances between the $$3$$ points ordering is not conserved, that is:

$$ab < bc < ac \quad | \quad a'b' < a'c' < b'c'$$

This example is consistent with the quoted statement since in the non-normalised case the $$\Delta x_2$$ differences dominate in the distance calculation.

Is there any other reason that justifies feature scaling? Additionally, are there any cases that normalization can decrease the performance?