# Does NSP task corrupt context during pre-training?

During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by sentence A’s context.

If we use MLM + NSP, our input will be [CLS] SentenceA [SEP] SentenceB [SEP]. In this case, the masked token in sentence A will be predicted by both contexts of sentence A and sentence B. Words in sentence B will contribute the final state of the masked token, even if they never appeared together.

For positive sampling sentence A and sentence B might share somehow a common context since they appear in the same document, but for negative sampling, they even don’t share that common context.

Does NSP corrupt context during pre-training? How BERT does deal with this corruption in pre-training?

I've tried to understand how the NSP task affects predictions in the MLM task, I've used BertForMaskedLM which is basically Bert for pre-training without an NSP head. So it will only calculate the loss for MLM. If I used BertForPreTraining to train both MLM+NSP, my input had to be [CLS] SentenceA [SEP] SentenceB [SEP]. For the MLM task, it would be passed to the MLM head, which is identical to BertForMaskedLM so it means they would handle the input in the same way.

Sentence A: The man went to [MASK] store Sentence B: He bought a gallon of milk

from transformers import BertTokenizer, BertForMaskedLM
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

## [CLS] the man went to [MASK] store [SEP] he bough a gallon of milk [SEP]
x = tokenizer.encode_plus("The man went to [MASK] store", "He bought a gallon of milk", return_tensors="pt")

y = model(x["input_ids"], x["token_type_ids"])
print(tokenizer.decode(word_prediction))

## Output: when, then, once, and, so

## [CLS] the man went to [MASK] store [SEP]
x = tokenizer.encode_plus("The man went to [MASK] store", return_tensors="pt")