2
$\begingroup$

In Roberta, I'm not sure if the model use BPE or byte-level BPE tokenization, are these techniques different or the same ? Can someone explain ? Thanks

$\endgroup$

1 Answer 1

4
$\begingroup$

Byte-level BPE is a subtype of BPE that uses bytes instead of characters as basic token component.

In the character-level BPE, the vocabulary is composed of sequences of characters that appear frequently together found by means of the BPE algorithm, but also includes all the individual characters seed in the training data. This allows to resort to character-by-character word composition for unseen words. This, however, poses two problems: 1) if at inference time we bump into unseen characters, the tokenizer will not be able to represent them, and 2) there are a lot of unicode characters, which needs a lot of room in the vocabulary to be represented, especially taking into account that a reasonable vocabulary size should be around 50k tokens.

In the byte-level BPE, the vocabulary is not created based on characters, but based on bytes. For that, characters are decomposed as the associated sequence of bytes in the unicode representation. This removes completely the possibility of unseen characters and saves space when representing individual elements (i.e. bytes) in the vocabulary.

In this article that studies the application of byte-level BPE to machine translation, you can find more details about this kind of BPE.

$\endgroup$
4
  • $\begingroup$ I want to ask character-level BPE is the same as regular BPE, how many variations of BPE are there ? $\endgroup$
    – user159173
    Commented Feb 4 at 18:05
  • 2
    $\begingroup$ Character-level BPE was the original BPE. However, nowadays many LLM's use byte-level BPE so when someone says just "BPE" I would not assume that it's character-level, I would assume lack of information about whether it's character-level or byte-level. $\endgroup$
    – noe
    Commented Feb 4 at 18:15
  • $\begingroup$ So does Roberta use character-level BPE or byte-level BPE ? $\endgroup$
    – user159173
    Commented Feb 5 at 21:00
  • $\begingroup$ Byte-level BPE. This is stated in section 4.4 of the article: "Following Radford et al. (2019), we instead consider training BERT with a larger byte-level BPE vocabulary". $\endgroup$
    – noe
    Commented Feb 5 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.