In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned:

The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special consideration given to partial word pieces.

And in the RoBERTa paper, section '4.4 Text Encoding' it is mentioned:

The original BERT implementation (Devlin et al., 2019) uses a character-level BPE vocabulary of size 30K, which is learned after preprocessing the input with heuristic tokenization rules.

I appreciate if someone can clarify why in the RoBERTa paper it is said that BERT uses BPE?


BPE and word pieces are fairly equivalent, with only minimal differences. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning.

Therefore, I understand that the authors of RoBERTa take the liberty of using BPE and wordpieces interchangeably.

  • $\begingroup$ Thank you @ncasas. Does that mean technically I can pre-train and fine-tune a RoBERTa based model with a WordPiece tokenizer? (I am using the Huggingface liblary) $\endgroup$
    – Adel
    Dec 12 '20 at 10:48
  • 1
    $\begingroup$ Yes, you can pre-train a RoBERTa with a wordpiece tokenizer, and then fine-tune it with the same wordpiece tokenizer. $\endgroup$
    – noe
    Dec 12 '20 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.