0
$\begingroup$

During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by sentence A’s context.

If we use MLM + NSP, our input will be [CLS] SentenceA [SEP] SentenceB [SEP]. In this case, the masked token in sentence A will be predicted by both contexts of sentence A and sentence B. Words in sentence B will contribute the final state of the masked token, even if they never appeared together.

For positive sampling sentence A and sentence B might share somehow a common context since they appear in the same document, but for negative sampling, they even don’t share that common context.

Does NSP corrupt context during pre-training? How BERT does deal with this corruption in pre-training?

I've tried to understand how the NSP task affects predictions in the MLM task, I've used BertForMaskedLM which is basically Bert for pre-training without an NSP head. So it will only calculate the loss for MLM. If I used BertForPreTraining to train both MLM+NSP, my input had to be [CLS] SentenceA [SEP] SentenceB [SEP]. For the MLM task, it would be passed to the MLM head, which is identical to BertForMaskedLM so it means they would handle the input in the same way.

Sentence A: The man went to [MASK] store Sentence B: He bought a gallon of milk

from transformers import BertTokenizer, BertForMaskedLM
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

## [CLS] the man went to [MASK] store [SEP] he bough a gallon of milk [SEP]
x = tokenizer.encode_plus("The man went to [MASK] store", "He bought a gallon of milk", return_tensors="pt")

masked_index = torch.where(x["input_ids"] == tokenizer.mask_token_id)[1].tolist()[0]

y = model(x["input_ids"], x["token_type_ids"])
for prediction in y[0][:, masked_index].topk(5).indices.tolist()[0]:
    print(tokenizer.decode(word_prediction))

## Output: when, then, once, and, so

## [CLS] the man went to [MASK] store [SEP]
x = tokenizer.encode_plus("The man went to [MASK] store", return_tensors="pt")
masked_index = torch.where(x["input_ids"] == tokenizer.mask_token_id)[1].tolist()[0]

y = model(x["input_ids"], x["token_type_ids"])
for prediction in y[0][:, masked_index].topk(5).indices.tolist()[0]:
    print(tokenizer.decode(word_prediction))

## Output the, a, his, another, her

According to the code example above, can we say NSP corrupts the context and affects MLM tasks badly even if the sentences are sequential?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.