When training a neural net, I understand the value in normalising the input data to have mean = 0 and stdev = 1 (standardising the data). But I often see people make the data even more "normal" by transforming the data so its shape better matches the normal distribution, not just the metrics mean and stdev.
I am hoping someone can help me understand this, I've tried below to illustrate where my lack of understanding comes in.
If I have a feature that I know follows a Weibull distribution, its density function could look something like this (lambda = 1, k = 1.5):
(I've just binned the data I have to show the density shape).
I would then standardise the data so that the mean = 0 and stdev = 1, and we would get a density that looks like:
This is where I would stop with pre-processing and where I become confused.
Why is it often recommended to go one step further and transform the distribution to have the shape of the normal distribution. Doing this results in a density that looks like this:
Intuitively to me, it seems like we lose some information by changing the distribution, if anyone could explain where my intuition is wrong, I'd greatly appreciate that.
Also, I feel like there is a big downside to changing the distribution of the feature that is that probably just more of my misunderstanding. The downside I see to this is that if we know the data skews to the low values (like the Weibull with lambda = 1, k = 1.5) does, surely we want to train the model more on these lower values that the model, once trained and live, will see more of. Don't we care more about how the model performs on these lower values than on the higher values?
