Why shouldn't we mask [CLS] and [SEP] in preparing inputs for a MLM?

Question

I know that MLM is trained for predicting the index of MASK token in the vocabulary list, and I also know that [CLS] stands for the beginning of the sentence and [SEP] telling the model the end of the sentence or another sentence will come soon, but I still can't find the reason for unmasking the [CLS] and [SEP].

Here is the situation that I imagine: We have a sentence pair like s1/s2, we input its input_ids into the model as the form like "101 xxx 102 yyy 102" and then I think we can ask model predict the token at the middle of the 'xxx' and 'yyy'(namely the first 102), so we can mask the token as 103 which means the MASK token.

I think the imagination is reasonable, could anyone give me a key?

Answer 1

In practice, nothing is preventing one from doing what you propose, masking and predicting the [CLS] or the [SEP] token. But the important question is why the model would need to learn about unmasking these tokens.

My understanding is that language models like BERT are pretrained for giving them a better understanding of the language. Then they can be finetuned for any downstream task. But [CLS] and [SEP] tokens are not part of the language and we add them for our convenience. You do not need to learn about them for getting a better "understanding of the language". Learning to predict a masked [SEP] token may not bring any additional performance improvement to the models.

[CLS] is used as a representative of the whole input text sequence which is then used for classification tasks usually. It should not be treated the same way as other tokens because it is serving a different purpose. Similar reasoning may be used for [SEP] tokens.