# In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that :

• the positional encoding is summed to the word embeddings
• rather than concatenated to it.

http://jalammar.github.io/images/t/transformer_positional_encoding_example.png

Based on the graphs I have seen wrt what the encoding looks like, that means that :

• the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot,
• while there is also a large amount of positions in the embedding that are only slightly affected by the positional encoding (when you move further towards the end).

https://www.tensorflow.org/beta/tutorials/text/transformer_files/output_1kLCla68EloE_1.png

So, why not instead have smaller word embeddings (reduce memory usage) and a smaller positional encoding retaining only the most important bits of the encoding, and instead of summing the positional encoding of words keep it concatenated to word embeddings?

• I was also curious about this, have you figured it out? Feb 1, 2020 at 14:34
• @LeeMJ: No, I did not. Feb 4, 2020 at 11:26
• Have you figured it out now? May 25, 2020 at 18:16
• Is anyone aware of any papers where they tried concatenation instead of adding? Feb 24, 2021 at 16:29
• @keith-johnson Not per se, but Google T5 does use a different approach, where position is encoded separately. Since there is a lot about Google T5 you can maybe also check this other paper that builds on top of T5 and tweaks its positional encoding some more: arxiv.org/pdf/2102.09550.pdf Feb 24, 2021 at 19:48

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have to define two dimensionalities, $$d_t$$ for the token and $$d_p$$ for the position, with the total dimensionality $$d = d_t + d_p$$, so $$d>d_t$$ and $$d>d_p$$. We would be decreasing the total size we devote to tokens in favor of positional information.

However, adding them together is potentially a super case of the concatenation: imagine that there is an ideal split of $$d$$ into $$d_t$$ and $$d_p$$ in terms of minimizing the loss; then, the training could converge to position vectors that only take $$d_t$$ elements, making the rest zero, and the positions were learned and happened the same, taking the complementary $$d_p$$ elements and leaving the rest to zero.

Therefore, by adding them, we leave the optimization of the use of the $$d$$ dimensions to the optimization process, instead of assuming there is an optimal partition of the vector components and setting a new hyperparameter to tune. Also, the use of the vector space is not restricted by a hard split in the vector components, but takes the whole representation space.

• This would make sense for learned positional encoding. What about the sine/cosine encoding? Does it just rely on the fact that nothing much is happening in dimensions beyond the first few?
– max
Feb 24, 2021 at 5:56
• While the equivalence of concatenation and addition may only apply to learned positional encoding, I think that the general optimization of the representation space does apply to fixed encodings as well (although the optimization only happens in the token embeddings). I don't think the picture is correct (it has changed in the referenced tutorial)
– noe
Feb 24, 2021 at 8:24
• maybe a stupid question, but why this addition doesn't spoil the embedding - like we had word king, add this pattern and receive slave ? Feb 25, 2021 at 21:39
• If such a thing would happen, the final loss would be bad. The training aims at improving the loss, and therefore prevents that situation from happening.
– noe
Feb 25, 2021 at 21:45
• You mentioned adding them, concatenating them. How about other ways like vector-matrix multiplication? It's a linear transformation. Theroretically, can this be used to integrate positional information as well? Is it a bad approach because of too many parameters (one matrix per position)? Dec 3, 2022 at 9:48

the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot

This confused me very much at first because I was thinking of the model using a pre-trained word embedding. And then an arbitrary initial chunk of that embedding gets severely tampered with by the positional encoding.

However, in the original transformer model at least, the embedding was trained from scratch, so this does not apply. An initial chunk of the overall embedding will be used for positional information, and the rest will be used for word information.

This still doesn't explain why we use this method instead of concatenation -- see the other answers for that -- but it does explain why the method isn't crazy.

That said, it may be that the method works well even with pre-trained word embeddings, I don't know. If so, it's hard to explain.

The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the word embedding; however, it is not actually. Adding new dimensions to each embedding increases the dimensionality of the problem. On the other hand, please note that the added positional embedding is (almost) static, as shown in this image for a 2D positional embedding:

The added positional embeddings are the same for all the inputs, and the transformer can separate the positional information from the actual word embedding through the training process. Therefore, the positional embedding doesn't mess with the word embedding information, and adding them is a more efficient way of adding the positional information that concatenates them.

It is been a while, but I think anyone ending up here might also be interested in the reading of the following paper:

What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding (Yu-An Wang, Yun-Nung Chen)

https://www.aclweb.org/anthology/2020.emnlp-main.555

The following is conjecture, not fact.

If you look at how much each scalar in the the positional embedding vector changes as a function of position... you'll find that many of the scalars barely change at all. You can visualize this with any positional embedding plot, where the x axis is usually the [512] length of the vector, and the y axis is the position of the token.

For example, this image is from Jay Alammar's well regarded "The Illustrated Transformer"

Let's try to do this mathematically as well. The implementation of PE's that Jay references is at this Google GitHub repo:

https://github.com/tensorflow/tensor2tensor/tree/23bd23b9830059fbc349381b70d9429b5c40a139

Running the function on a PE/WE of length 512 and max sentence length of 128, let's look at how much the final value in the vector actually changes from the first position, to the 64th position, to the final position. Answer: not much.

print(signal[0, 0, -1])
print(signal[0, 63, -1])
print(signal[0, 127, -1])

tf.Tensor(1.0, shape=(), dtype=float32)
tf.Tensor(0.99998015, shape=(), dtype=float32)
tf.Tensor(0.99991935, shape=(), dtype=float32)


Ditto for a value 16 steps away from the final location:

print(signal[0, 0, -16])
print(signal[0, 63, -16])
print(signal[0, 127, -16])

tf.Tensor(1.0, shape=(), dtype=float32)
tf.Tensor(0.9984067, shape=(), dtype=float32)
tf.Tensor(0.9935305, shape=(), dtype=float32)


I saw elsewhere that BERT's WEs are typically roughly the range of [-2, 2], so adding a 0.007 delta from the PE would not move the WE very much at the -16th position.

So what I think is probably happening is that only ~256 of the PE vector's values are actually moving around as a function of the position... the rest are ~constant. Then the learned WE (Transformers don't use prelearned WE like word2vec or glove), figures out to only use the other ~256 elements. So really... it's conceptually a concat.

notebook here