How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder

Question

Bert has two types of tasks that it uses to learn contextual word embeddings:

1. Masked word prediction
2. Next sentence prediction

I have read the paper and even there the training details are a little fuzzy or, dont make sense to me

To quote the paper:

To generate each training input sequence, we sample two spans of text from the corpus, which we refer to as “sentences” even though they are typically much longer than single sentences (but can be shorter also). The first sentence receives the A embedding and the second receives the B embedding. 50% of the time B is the actual next sentence that follows A and 50% of the time it is a random sentence, which is done for the “next sentence prediction” task. They are sampled such that the combined length is ≤ 512 tokens. The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special consideration given to partial word pieces.

Question: If 50% of our training examples are sentence (in Bert sense) followed by another unrelated sentence and we run it as one sequence through the transformer and the objective is to find context-specific embeddings then how does the objective of masked language modelling make sense in these cases? Our second sentence is the wrong context here. Masked language modelling makes sense only if the second sentence is the accurate entailment which is the case only in 50% of the cases

Answer 1

First, note that the purpose of next sentence prediction objective is not to contribute to the contextual embeddings part, but to allow other downstream tasks like sentence classification and textual entailment.

The contextual embedding signal comes from the part of the loss that predicts the masked token.

The model receives information to distinguish the two sentences. As you can see from your quoted paragraph (i.e. The first sentence receives the A embedding and the second receives the B embedding.) and from figure 2 from the BERT article itself:

Therefore, the model knows where the two sentences are (with the different $E_A$, $E_B$ embeddings and the $E_{[SEP]}$ token. Then, it is up to the model to learn to only use each sequence to predict the masked tokens to avoid using information from a segment that is not related, or even to use information from both segments but only if the model thinks they belong together.

Of course, this is not a guarantee that the model does not mix information from an unrelated sentence in the prediction.

Actually, in the derived model RoBERTa, created by different authors, the next sentence prediction objective was completely removed.