I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-Vocabulary (OOV) words. Can someone please help me explain how WordPiece tokenization is actually done, and how it handles effectively helps to rare/OOV words?

  • 3
    $\begingroup$ This question has been answered here. I'm copying the answer here as well. WordPiece and BPE are two similar and commonly used techniques to segment words into subword-level in NLP tasks. In both cases, the vocabulary is initialized with all the individual characters in the language, and then the most frequent/likely combinations of the symbols in the vocabulary are iteratively added to the vocabulary. Consider the WordPiece algorithm from the [original paper](static.googleusercontent.com/media/research.google.com/en//pubs/… $\endgroup$
    – Harman
    Apr 2, 2019 at 12:35
  • 2
    $\begingroup$ The URL is broken. Here the corrected one. $\endgroup$
    – Laura
    Aug 22, 2019 at 14:52
  • $\begingroup$ Here you go... youtube.com/watch?v=zJW57aCBCTk | I've recently started following this guy. He has a very nice way of explaining such concepts. $\endgroup$
    – Aditya
    Jan 9, 2020 at 7:54

1 Answer 1


WordPiece tokenization is a type of subword tokenization. Subword tokenization is a technique for splitting words into smaller units, called subwords, that are still meaningful. This is in contrast to traditional word tokenization, which simply splits words on whitespace or punctuation.

WordPiece tokenization works by first creating a vocabulary of subwords. This vocabulary is created by iteratively merging the most common subwords together until the desired vocabulary size is reached. The merging process is done in a way that minimizes the information loss.

When a word is tokenized using WordPiece, it is first pre-tokenized into words by splitting on whitespace and punctuation. Then, each word is tokenized into subwords using the vocabulary. If a word is not in the vocabulary, it is split into subwords using the merging process.

WordPiece tokenization is effective for handling rare/OOV words because it allows the model to learn the meaning of words even if they are not in the vocabulary. This is because the model can learn the meaning of a word from the subwords that make up the word.

For example, the word "playing" is not in the vocabulary of a WordPiece model. However, the model can learn the meaning of the word by learning the meaning of the subwords "play" and "ing".

WordPiece tokenization is a powerful technique for handling rare/OOV words in NLP models. It is used in many popular models, such as BERT, DistilBERT, and Electra.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.