4
$\begingroup$

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should not normalise it. However, isn't this always the case? For example, if we have age and salary, isn't salary always going to be greater than age? Wouldn't normalising these values lose possible significant information?

Also, should you always normalise the entire dataset, or can one simply normalise a number of specific columns?

$\endgroup$
2
$\begingroup$

Normalisation is performed to balance weights and make parameters universal. In the cases you mention this is interval normalisation (thus numbers, thus a set that has standard order and distance metric). Just for the contrast, string normalisation might involve stemming, converting to uppercase or a number of other techniques, depending on the context. Since scaling (multiplying by the same factor) is a linear operation, it preserves order and magnitude relations, among other properties. Thus, in the general case, normalisation does not loose information about relationships within the parameter/dimension.

If we use a simple clustering algorithm for the sake of the example, consider that age might have a range (roughly) of (0-100) and salaries could be something like the interval (1-10^6). A naive approach would be not to perform normalisation and to use the default Euclidean distance metric. However, variances of salaries of up to $100 are negligible. Thus your salary information would dominate your age information, resulting distances between data points (and thus clusters) being mostly determined by the salary only. The secondary role of age would become negligible for the clustering. However, if we normalise the two to the range of (0-1), we might get somewhat comparable scales and the factor influence would be incomparably smaller (while still to be considered).

Due to the last remark in the brackets, most advanced machine learning algorithms (including neural networks, support vector machines) actually handle different scales. When you use these, you don't need to normalise data, because the algorithm learns normalisation along with other properties of the data.

The decision of when to normalise your data depends on your problem context and on the approach you are planning to take. Let's say that normalising data makes it easier from the first view to be able to tell whether a value is low or high.

$\endgroup$
0
3
$\begingroup$

Normalisation is a very blurry concept. It is quite misunderstood most of the times. I will take the specific case of Neural Nets and explain it:

  • Purpose: Normalisation is done so that the Neural Network weights converge faster. In CNN's and Deep Neural Nets this is of particular help especially in CNN this helps to prevent exploding/vanishing gradients.

The most common explanation for normalisation I have come across is that if you have 2 features, one of them has a significant larger scale than the other e.g house price and house area then the feature with a larger scale will dominate the output. This is quite incorrect according to me, since when you back-propagate through the Neural Net the weight updates are directly proportional to the activations, so larger activation means larger feedback and hence weights get reduced faster and become smaller until w1*house price = w2*house area approximately this relation holds true. Yes it will lead to more oscillations (intuitionally since learning rate also becomes multiplied with a larger scale) but it will ultimately probably converge.

So the best 3 reasons for using normalisation are:

  • If scale of a feature is large the weights connected to that feature will have larger oscillations resulting in slower convergence and if deep NN is used probably no convergence, whereas normalisation helps in making the values small -1 to 1 so the gradient updates are also small resulting in faster convergence.
  • The best intuition for normalisation can be found from this video of Stanford and its subsequent video. Since we know weight updates is directly proportional to inputs it will also take the sign of the inputs (or exact opposite sign) always. Now we know house price and area is always positive (in our Universe at least!). So the weight updates will always have definite sign either both positive or both negative (for error being propagated from a single next layer node). But the weight updates may have a optimal direction in the 4th quadrant, so the weight updates will follow a zig zag pattern which will result in loss of efficiency.
  • Finally when you are dealing with Deep Neural Nets like CNN if you do not normalise pixels it will result in exponentially large/vanishing gradients. Since generally softmax/sigmoid is used in the last layer, it squashes the outputs. If you have a large output, generally due to un-normalized data, it'll result in wither exact 0 or exact 1 output, which is fed into a log function and BAM! overflow. The error becomes inf or NaN in python. So inf error means exploding gradients and NaN means gradient cannot be calculated. So you fail to train from the start and it continues inf is generally followed by NaN which will continue to eternity. This can probably be remedied by using higher floating point precision but it will result in higher memory and processor consumption ultimately inefficiency.

TL;DR: Normalisation is used for faster weight convergence. Issues faced by un-normalised data are larger weight oscillations, weight updation in non optimal direction, overflow in precision in Deep Neural Nets.

$\endgroup$
5
  • $\begingroup$ Thanks. So in other words, we normalise for the sake of model execution performance and not accuracy, right? $\endgroup$
    – David
    Aug 21 '18 at 13:28
  • $\begingroup$ @DavidFarrugia model performance is directly related to accuracy...if it converges accuracy must and should increase $\endgroup$
    – DuttaA
    Aug 21 '18 at 13:29
  • $\begingroup$ what is exploding & vanishing gradients? Can you share some cool resource that explains it really well @DuttaA $\endgroup$ Jul 18 '19 at 17:13
  • 1
    $\begingroup$ @AbhimanyuAryan exploding gradients mean that the gradients become very large, say in sigmoid you are supposed to predict 1 but you predict 0, which will give a large loss and hence a very large gradient if the loss is not scaled properly and it becomes larger and larger as it goes to previous layers. $\endgroup$
    – DuttaA
    Jul 18 '19 at 23:23
  • 1
    $\begingroup$ @AbhimanyuAryan In vanishing gradient, you have made an almost accurate prediction so the loss is low, the gradients are small, and is it backpropagates through the NN the gradient becomes smaller and smaller till it vanishes (effective in case of sigmoid and tanh only). You can visualize this by doing the maths by taking a few sigmoid layers. I would suggest you to do a quick google search, and I think you'll find quite good illustrations of these problems, just make sure it is from authorative sources like good colleges or good instructors. $\endgroup$
    – DuttaA
    Jul 18 '19 at 23:24
1
$\begingroup$

Both matpo and DuttaA give good technical answers, so I'll add an easy to remember phrase that covers a large swatch of the reason to normalize: apples and oranges.

When you have variables measured in different units, you should always normalize unless there is a compelling reason not to.

$\endgroup$
1
  • $\begingroup$ Thanks. Does the normalising method aslo affect? So should one take care of when to use z-score or tangent values for instance? $\endgroup$
    – David
    Aug 22 '18 at 5:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.