5
$\begingroup$

What's the general tradeoff between choosing BPE vs WordPiece Tokenization? When is one preferable to the other? Are there any differences in model performance between the two? I'm looking for a general overall answer, backed up with specific examples.

$\endgroup$
2
  • 2
    $\begingroup$ You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very fast C++ implementation of BPE. You can find fast Rust implementations of both in Hugginface's tokenizers. $\endgroup$ – noe Jun 2 '20 at 15:13
  • $\begingroup$ I made an answer out of my previous comment. $\endgroup$ – noe Mar 25 at 2:09
4
$\begingroup$

(This answer was originally a comment)

You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very fast C++ implementation of BPE. You can find fast Rust implementations of both in Hugginface's tokenizers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.