Understanding data normalisation

Question

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should not normalise it. However, isn't this always the case? For example, if we have age and salary, isn't salary always going to be greater than age? Wouldn't normalising these values lose possible significant information?

Also, should you always normalise the entire dataset, or can one simply normalise a number of specific columns?

Answer 1

Normalisation is performed to balance weights and make parameters universal. In the cases you mention this is interval normalisation (thus numbers, thus a set that has standard order and distance metric). Just for the contrast, string normalisation might involve stemming, converting to uppercase or a number of other techniques, depending on the context. Since scaling (multiplying by the same factor) is a linear operation, it preserves order and magnitude relations, among other properties. Thus, in the general case, normalisation does not loose information about relationships within the parameter/dimension.

If we use a simple clustering algorithm for the sake of the example, consider that age might have a range (roughly) of (0-100) and salaries could be something like the interval (1-10^6). A naive approach would be not to perform normalisation and to use the default Euclidean distance metric. However, variances of salaries of up to $100 are negligible. Thus your salary information would dominate your age information, resulting distances between data points (and thus clusters) being mostly determined by the salary only. The secondary role of age would become negligible for the clustering. However, if we normalise the two to the range of (0-1), we might get somewhat comparable scales and the factor influence would be incomparably smaller (while still to be considered).

Due to the last remark in the brackets, most advanced machine learning algorithms (including neural networks, support vector machines) actually handle different scales. When you use these, you don't need to normalise data, because the algorithm learns normalisation along with other properties of the data.

The decision of when to normalise your data depends on your problem context and on the approach you are planning to take. Let's say that normalising data makes it easier from the first view to be able to tell whether a value is low or high.

Answer 2

Normalisation is a very blurry concept. It is quite misunderstood most of the times. I will take the specific case of Neural Nets and explain it:

• Purpose: Normalisation is done so that the Neural Network weights converge faster. In CNN's and Deep Neural Nets this is of particular help especially in CNN this helps to prevent exploding/vanishing gradients.

The most common explanation for normalisation I have come across is that if you have 2 features, one of them has a significant larger scale than the other e.g house price and house area then the feature with a larger scale will dominate the output. This is quite incorrect according to me, since when you back-propagate through the Neural Net the weight updates are directly proportional to the activations, so larger activation means larger feedback and hence weights get reduced faster and become smaller until w1*house price = w2*house area approximately this relation holds true. Yes it will lead to more oscillations (intuitionally since learning rate also becomes multiplied with a larger scale) but it will ultimately probably converge.

So the best 3 reasons for using normalisation are:

• If scale of a feature is large the weights connected to that feature will have larger oscillations resulting in slower convergence and if deep NN is used probably no convergence, whereas normalisation helps in making the values small -1 to 1 so the gradient updates are also small resulting in faster convergence.
• The best intuition for normalisation can be found from this video of Stanford and its subsequent video. Since we know weight updates is directly proportional to inputs it will also take the sign of the inputs (or exact opposite sign) always. Now we know house price and area is always positive (in our Universe at least!). So the weight updates will always have definite sign either both positive or both negative (for error being propagated from a single next layer node). But the weight updates may have a optimal direction in the 4th quadrant, so the weight updates will follow a zig zag pattern which will result in loss of efficiency.
• Finally when you are dealing with Deep Neural Nets like CNN if you do not normalise pixels it will result in exponentially large/vanishing gradients. Since generally softmax/sigmoid is used in the last layer, it squashes the outputs. If you have a large output, generally due to un-normalized data, it'll result in wither exact 0 or exact 1 output, which is fed into a log function and BAM! overflow. The error becomes inf or NaN in python. So inf error means exploding gradients and NaN means gradient cannot be calculated. So you fail to train from the start and it continues inf is generally followed by NaN which will continue to eternity. This can probably be remedied by using higher floating point precision but it will result in higher memory and processor consumption ultimately inefficiency.

TL;DR: Normalisation is used for faster weight convergence. Issues faced by un-normalised data are larger weight oscillations, weight updation in non optimal direction, overflow in precision in Deep Neural Nets.

Answer 3

Both matpo and DuttaA give good technical answers, so I'll add an easy to remember phrase that covers a large swatch of the reason to normalize: apples and oranges.

When you have variables measured in different units, you should always normalize unless there is a compelling reason not to.