When training a neural net, I understand the value in normalising the input data to have mean = 0 and stdev = 1 (standardising the data). But I often see people make the data even more "normal" by transforming the data so its shape better matches the normal distribution, not just the metrics mean and stdev.

I am hoping someone can help me understand this, I've tried below to illustrate where my lack of understanding comes in.

If I have a feature that I know follows a Weibull distribution, its density function could look something like this (lambda = 1, k = 1.5):

enter image description here

(I've just binned the data I have to show the density shape).

I would then standardise the data so that the mean = 0 and stdev = 1, and we would get a density that looks like: enter image description here

This is where I would stop with pre-processing and where I become confused. Why is it often recommended to go one step further and transform the distribution to have the shape of the normal distribution. Doing this results in a density that looks like this: enter image description here

Intuitively to me, it seems like we lose some information by changing the distribution, if anyone could explain where my intuition is wrong, I'd greatly appreciate that.

Also, I feel like there is a big downside to changing the distribution of the feature that is that probably just more of my misunderstanding. The downside I see to this is that if we know the data skews to the low values (like the Weibull with lambda = 1, k = 1.5) does, surely we want to train the model more on these lower values that the model, once trained and live, will see more of. Don't we care more about how the model performs on these lower values than on the higher values?


1 Answer 1


The objective of standardization is that the values of features must be comparable. for example in housing price prediction the number of rooms and the size of house are very different and we need to normalise before it can be fed into network . if you do not normalise the impact of size of house will be much more than number of room . you are right that we need to bring the range of values in comparable bracket however it is not at all necessary to make it fit like a normal distribution . you are also right that it will lead to loss of information hope this helps

  • $\begingroup$ You can use min-max scaler as well $\endgroup$
    – amol goel
    Aug 14, 2022 at 5:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.