It's a common practice to normalize inputs to the neural Network.

Let's assume we have a vector of activations.

One of techniques, the Layer Normalization simply looks at the vector's components, re-centers this activated vector from $\mu$ to zero, then divides by the standard deviation $\sigma$

How is it then possible to distinguish activations [1,2,3,4] from [4,5,6,7] if both will be re-centered to the same vector [-1.5, -0.5, 0.5, 1.5f] and then divided by std deviation? Also, I can see such a problem when merely normalizing the input-state-vectors for any Neural Net.


there seems to be a hint in first half of page 4 in the paper, however due to my weakness in Maths I can't comprehend it :(

Edit after accepted the answer:

guys, don't forget that Layer Norm (and Batch norm) both have Gain and Bias terms. If the network performs poorly, the gain is tweaked to undo division-by-std-deviation, and tweaking the bias is used to undo the shift (the re-centering). This allows some neurons to indeed pay attention to scaling and shifting, when really required.


1 Answer 1


The math you are talking about is in equation (7):

Let x′ be a new data point obtained by re-scaling x by δ. Then we have, enter image description here

It is easy to see re-scaling individual data points does not change the model’s prediction under layer normalization. Similar to the re-centering of the weight matrix in layer normalization, we can also show that batch normalization is invariant to re-centering of the dataset.

It proves that the prediction over x' would be the same as in x. So what you say is right, they are indistinguishable.

And I think this is what we want because improves generalization in predictions!

For example in image classification, you are trying to detect orangutangs, if an image I represent an orangutang, ( I x 5)+2 still is an orangutang!

  • $\begingroup$ Thanks! So is it then only for classification tasks? $\endgroup$
    – Kari
    Sep 28, 2018 at 9:11
  • $\begingroup$ No I guess is the same even in regression! $\endgroup$ Sep 28, 2018 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.