# When should I normalize data?

I often see that numeric values in machine learning is scaled to 0-1 range. Why is it better?

I have some temperature values in my training set. What if I will have some values to predict that will be outside training set values?

I mean that eg in training set I will have range of temperatures like 5-20 and MinMaxScaler will fit to these values, and then I will have 25 to predict.

• Pre-processing is done before it is fed as input. If the model is learned correctly then, I guess you should get the correct output for any test input. – Siladittya Jun 24 '18 at 14:39
• It depends what's the Algorithm your are using.. like decision trees don't need normalisation but neural nets need it – Aditya Jun 25 '18 at 7:00

As @Daniel Chepenko pointed out, there are models that are robust w.r.t. feature transformations (like Random Forest). But for model which made operations on the features (like Neural Networks), usually you need to normalize data for three reasons:

1) Numerical stability: computers cannot represent every number, because the electronic which make them exist deals with binaries (zeros and ones). So they use a representation based on Floating Point arithmetic. In practice, this means that the numerical behavior in the range [0.0, 1.0] is not the same of the range [1'000'000.0, 1'000'001.0]. So having two features that have very different scales can lead to numerical instability, and finally to a model unable to learn anything.

2) Control of the gradient: imagine that you have a feature that spans in a range [-1, 1], and another one that spans in a range [-1'000'000, 1'000'000]: the weights associated to the first feature are much more sensitive to small variations, and so their gradient will become much more variable in the direction described by that feature. This can lead to other instabilities: some values of learning rate (LR) can be too small for one feature (and so the convergence will be slow) but too big for the second feature (and so you jump over the optimal values). And so, at the end of the training process you will have a sub-optimal model.

3) control of the variance of the data: if you have skewed features, and you don't transform them, you risk that the model will simply ignore the elements in the tail of the distributions. And in some cases, the tails are much more informative than the bulk of the distributions.

Data is usually normalized to make sure that all of your features on roughly the same scale and that the units you measure your data in do not make a difference to the model you fit in the end.

If you have data in the range 5-20 in the training set then in the test set your 25 will be mapped to 1.33 by the scaling (this is why the Scaler is fit to the training data, so you get a consistent mapping across training and test data). This is not a problem at all since your model doesn't really depend on your data being in [0,1].

It is actually depends on the algorithm you are using. For example, for random forests the ranges don't matter, since one feature is never compared in magnitude to other features. It's only the range of one feature that is split at each stage.

But on the other hand SVM or Logistic regression will probably do better if your features have roughly the same magnitude, unless you know apriori that some feature is much more important than others, in which case it's okay for it to have a larger magnitude.

I will try to explain it through an example.

Imagine that you have a problem of two attributes, temperature (Celsius) and length (mm). That problem requires the classification of the quality of long structural metal beams, based on the changes of their temperature and length during summer days.

By "long" it means they can be up to 2 meters long, that is [0-2000]mm range. Keep in mind that metal rods extend/shrink due to temperature changes. The temperature in that particular location changes between [20-35]Celsius during summer.

Assume that you would like to cluster the hourly samples of length and temperature using K-means clustering. Euclidean distance is usually the preferred choice to measure the distance between the cluster centers and other samples, in every iteration of the algorithm. This means that rods that have similar temperature (+-1 degree Celsius) but big difference in length (1000mm), will be in different clusters; but this might be misleading.

For that reason, you should scale all the dimensions in [0,1] range, so that the clustering distance is unbiased from measurement units.

Keep in mind that different units in engineering problems need minmax scaling in general, to have features that contribute to the classification outcome in a fair fashion.

MinMax scaler is not the only way to scale. There is also the StandardScaler which basically does:

\begin{align} x &\sim \mathcal{N}(\mu, \sigma)\\ x' :&= \frac{x-\mu}{\sigma} \end{align}

This leads to $x' \sim \mathcal{N}(0, 1)$.