3
$\begingroup$

I read the implementation of BERT inputs processing (image below). My question is why the author chose to sum up three types of embedding (token embedding, positional embedding and segment embedding)? enter image description here

$\endgroup$

2 Answers 2

3
$\begingroup$

Because these are the pieces of information needed to accomplish the loss' tasks, that is, both masked language modeling (i.e. predicting the masked tokens) and next sentence prediction (i.e. predict whether the second segment followed the first segment in the original text). These are the specific reasons:

  • Token embeddings are needed to identify the word/subword being processed as input, as well as the token being masked.
  • Positional embeddings are needed because without them, the Transformer cannot distinguish the same token in different positions (unlike recurrent networks like LSTMs). For more details, you can refer to this answer.
  • Sentence embeddings are needed for the secondary task of the loss: next sentence prediction. They are needed to easily tell apart the different parts of the input. For more details, you can refer to this answer.

Also, note that a normal Transformer architecture already adds token embeddings and positional embeddings.

The reason why these embeddings are added up instead of e.g. concatenated can be found in this answer. Adding them up you basically are learning the optimal way of combining them, instead of fixing it a priori.

$\endgroup$
0
$\begingroup$

Author of the paper believe that he can just sum up three different features into one feature, and it worked because feedforward neural network learned to live with such perturbations in it's input perception.

Jokes aside. It is known to everyone, that if you change input order in a simple feedforward neural network, such as home price prediction, where at first input index you have number of bedrooms and at second input index you have age of that estate, you would not get correct output, because FNN's learned that at first index it expects number of bedrooms, not the age of the building.

I explained it more in this answer.

Indeed, why author left separation token, when there is already segment encoding? Where is ablation studies in all these papers?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.