I read the implementation of BERT inputs processing (image below). My question is why the author chose to sum up three types of embedding (token embedding, positional embedding and segment embedding)? enter image description here


1 Answer 1


Because these are the pieces of information needed to accomplish the loss' tasks, that is, both masked language modeling (i.e. predicting the masked tokens) and next sentence prediction (i.e. predict whether the second segment followed the first segment in the original text). These are the specific reasons:

  • Token embeddings are needed to identify the word/subword being processed as input, as well as the token being masked.
  • Positional embeddings are needed because without them, the Transformer cannot distinguish the same token in different positions (unlike recurrent networks like LSTMs). For more details, you can refer to this answer.
  • Sentence embeddings are needed for the secondary task of the loss: next sentence prediction. They are needed to easily tell apart the different parts of the input. For more details, you can refer to this answer.

Also, note that a normal Transformer architecture already adds token embeddings and positional embeddings.

The reason why these embeddings are added up instead of e.g. concatenated can be found in this answer. Adding them up you basically are learning the optimal way of combining them, instead of fixing it a priori.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.