While reviewing the Transformer architecture, I realized something I didn't expect, which is that :

  • the positional encoding is summed to the word embeddings
  • rather than concatenated to it.

positional encoding summed to word embedding


Based on the graphs I have seen wrt what the encoding looks like, that means that :

  • the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot,
  • while there is also a large amount of positions in the embedding that are only slightly affected by the positional encoding (when you move further towards the end).

graph shows positional encoding affects firsts logits a lot, last logits hardly not


So, why not instead have smaller word embeddings (reduce memory usage) and a smaller positional encoding retaining only the most important bits of the encoding, and instead of summing the positional encoding of words keep it concatenated to word embeddings?

  • $\begingroup$ I was also curious about this, have you figured it out? $\endgroup$ – Lee MJ Feb 1 at 14:34
  • $\begingroup$ @LeeMJ: No, I did not. $\endgroup$ – FremyCompany Feb 4 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.