I know that MLM is trained for predicting the index of MASK token in the vocabulary list, and I also know that [CLS] stands for the beginning of the sentence and [SEP] telling the model the end of the sentence or another sentence will come soon, but I still can't find the reason for unmasking the [CLS] and [SEP].

Here is the situation that I imagine: We have a sentence pair like s1/s2, we input its input_ids into the model as the form like "101 xxx 102 yyy 102" and then I think we can ask model predict the token at the middle of the 'xxx' and 'yyy'(namely the first 102), so we can mask the token as 103 which means the MASK token.

I think the imagination is reasonable, could anyone give me a key?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.