Bert has two types of tasks that it uses to learn contextual word embeddings:

  1. Masked word prediction
  2. Next sentence prediction

I have read the paper and even there the training details are a little fuzzy or, dont make sense to me

To quote the paper:

To generate each training input sequence, we sample two spans of text from the corpus, which we refer to as “sentences” even though they are typically much longer than single sentences (but can be shorter also). The first sentence receives the A embedding and the second receives the B embedding. 50% of the time B is the actual next sentence that follows A and 50% of the time it is a random sentence, which is done for the “next sentence prediction” task. They are sampled such that the combined length is ≤ 512 tokens. The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special consideration given to partial word pieces.

Question: If 50% of our training examples are sentence (in Bert sense) followed by another unrelated sentence and we run it as one sequence through the transformer and the objective is to find context-specific embeddings then how does the objective of masked language modelling make sense in these cases? Our second sentence is the wrong context here. Masked language modelling makes sense only if the second sentence is the accurate entailment which is the case only in 50% of the cases

  • $\begingroup$ For ease of access, please provide a link or proper citation to the quoted paper. $\endgroup$
    – lpounng
    Commented Nov 14, 2023 at 7:57
  • $\begingroup$ Noted and Done. $\endgroup$ Commented Nov 15, 2023 at 5:46

1 Answer 1


First, note that the purpose of next sentence prediction objective is not to contribute to the contextual embeddings part, but to allow other downstream tasks like sentence classification and textual entailment.

The contextual embedding signal comes from the part of the loss that predicts the masked token.

The model receives information to distinguish the two sentences. As you can see from your quoted paragraph (i.e. The first sentence receives the A embedding and the second receives the B embedding.) and from figure 2 from the BERT article itself:

enter image description here

Therefore, the model knows where the two sentences are (with the different $E_A$, $E_B$ embeddings and the $E_{[SEP]}$ token. Then, it is up to the model to learn to only use each sequence to predict the masked tokens to avoid using information from a segment that is not related, or even to use information from both segments but only if the model thinks they belong together.

Of course, this is not a guarantee that the model does not mix information from an unrelated sentence in the prediction.

Actually, in the derived model RoBERTa, created by different authors, the next sentence prediction objective was completely removed.

  • 1
    $\begingroup$ I understood the purpose of different losses however, I was confused about the MLM loss also being generated from the sequences where the next sentence was not an actual entailment. But, that part you have answered in the later half of your answer. Going through RoBERTa paper was also revealing. Thank you for your time and knowledge :) $\endgroup$ Commented Nov 15, 2023 at 5:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.