0
$\begingroup$

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned:

The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special consideration given to partial word pieces.

And in the RoBERTa paper, section '4.4 Text Encoding' it is mentioned:

The original BERT implementation (Devlin et al., 2019) uses a character-level BPE vocabulary of size 30K, which is learned after preprocessing the input with heuristic tokenization rules.

I appreciate if someone can clarify why in the RoBERTa paper it is said that BERT uses BPE?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

BPE and word pieces are fairly equivalent, with only minimal differences. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning.

Therefore, I understand that the authors of RoBERTa take the liberty of using BPE and wordpieces interchangeably.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you @ncasas. Does that mean technically I can pre-train and fine-tune a RoBERTa based model with a WordPiece tokenizer? (I am using the Huggingface liblary) $\endgroup$
    – Adel
    Dec 12, 2020 at 10:48
  • 1
    $\begingroup$ Yes, you can pre-train a RoBERTa with a wordpiece tokenizer, and then fine-tune it with the same wordpiece tokenizer. $\endgroup$
    – noe
    Dec 12, 2020 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.