# Layer normalization details in GPT-2

I've read that GPT-2 and other transformers use layer normalization before the self-attention and feedforward blocks, but I am still unsure exactly how the normalization works.

Let's say that our context size is 1024 tokens, the embedding size is 768 (so that each token and its subsequent hidden states are represented by vectors of size 768), and we are using 12 multi-attention heads. So in the diagram above, there are 1024 r's and each r has dimensionality 768.

For a given layer in the transformer, how many normalization statistics (sample mean and stdev) are computed? Do we do one normalization per token, for 12x1024 normalizations so that the feature values within each token have mean 0 and std 1? Or do we normalize the values for each feature across tokens, for 12x768 normalizations? Or do we normalize all the feature values for all the tokens together, for 12 normalizations? Do we compute separate normalizations for each context in the minibatch?

I'm also keen to understand intuitively why this normalization is desirable. Assuming that the scheme is to normalize the feature values within each token: let's say one of our tokens is a bland word like "ok" whereas another token is the word "hatred". I would expect that the representation of "hatred" would be spikier, with higher variance among the different feature values. Why is it useful to throw away this information and force the representation for "ok" to be just as spiky? On the other hand, if the normalization scheme is to normalize across feature values, so that if you take feature 1 from all of the tokens in our context, they will have zero mean and stdev 1, doesn't this throw away information when all of the words in our context are very negative, for example in the context "war violence hate fear"?

Separately, with layer normalization it seems like it is optional to re-scale the normalized values through learned bias and gain parameters. Does GPT-2 do this, or does it keep the values normalized to mean 0 and std 1?

The most standard implementation uses PyTorch's LayerNorm which applies Layer Normalization over a mini-batch of inputs. The mean and standard-deviation are calculated separately over the last certain number dimensions which have to be of the shape specified by normalized_shape argument. Most often normalized_shape is the token embedding size.

The paper "On Layer Normalization in the Transformer Architecture" goes into great detail about the topic. The paper proposes "the layer normalization plays a crucial role in controlling the gradient scales." Better behaved gradients help with training.

• Hi Brian. To clarify, if normalized_shape is equal to the token embedding size, this means that in the diagram above, each r_i vector will be normalized individually, so that the features in each word's representation will have zero mean and unit standard deviation? In other words, we will compute 12x1024 total normalizations. This seems different from the original Layer Norm implementation, which normalized across an entire layer. Separately, do you know whether bias and gain terms are fit in GPT-2's layer norm? Jan 31, 2021 at 22:23

As per the reference, Layer Normalization is applied 2 times per block (or layer). Once for the hidden states from the output of the attention layer, and once for the hidden states for the output from the feed-forward layer. However, it is

(For hugging-face implementation, you can check out class Block here)

For your example, normalization necessarily doesn't really does not change the order of spikes of various tokens, but changes the magnitude of the spikes. This has shown to increase the model training time, and improve the performance. (paper)

• Hi Ashwin, thanks. I understand where LayerNorm occurs in the architecture. I am asking about the computational details: which set of scalars exactly are used to compute each sample mean and standard dev, and how many such normalization statistics are computed in my example. This is not clear to me after reading the papers, and if it is obvious to you please be explicit about the details. Jan 30, 2021 at 16:47