As I understood, when training a neural network, it is preferable to have data with expectation of 0 and std. of 1.

Now if I have a feature with a Ratio distribution i.e., where median and expectation can differ much. If I apply Gaussian normalization - to subtract expectation and divide by std, I will get a distribution with again median different to expectation.

enter image description here

Should I be bothered by this anyway? I can easily move them to match by applying Log to it but I wanna know should I waste time on such details?


Normalising input data to a neural network is known to improve convergence properties, i.e. the model should converge quicker because the normalised data will not be so likely to produce huge or tiny gradients. Huge gradients make training erratic and less likely to converge (without applying other tricks), while tiny loss gradients lead to neuron death whereby certain neurons may get stuck, unable to get to an optimal weight value.

As you pointed out, normalising can be trivially simply to perform, so it might be worthwhile for both the final results and as a learning experience to try it out and compare your results.

You might also consider using other techniques such as batch normalisation, which inherently give you similar properties to normalising the entire dataset at the beginning.

  • $\begingroup$ I would just add to this answer that at an intuitive level, it is also claimed to help with treating different scales similarly, otherwise since the scales are different, one may have issues. $\endgroup$
    – PSub
    Jun 5 '18 at 21:45
  • $\begingroup$ That's a good point @PSub - thanks for complementing my answer. $\endgroup$
    – n1k31t4
    Jun 5 '18 at 21:52
  • $\begingroup$ Thanks! So answer is "try it out" :), ok, I will, I thought someone already tried... Thanks for pointing on other techniques, i'm new at this and every info is valuable! $\endgroup$ Jun 5 '18 at 21:58
  • $\begingroup$ People have tried it out, I can assure you ;) - the real answer is that it will depend a lot on your specific dataset. $\endgroup$
    – n1k31t4
    Jun 5 '18 at 22:01

Normalization rescales the values into a range of [0,1].

Standardization rescales data to have a mean (μ) of 0 and standard deviation (σ) of 1 (unit variance)

Both techniques are used for a different purpose, although both can change the distribution shape.

Check this link for more information.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.