How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

Question

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-Vocabulary (OOV) words. Can someone please help me explain how WordPiece tokenization is actually done, and how it handles effectively helps to rare/OOV words?

Answer 1

WordPiece tokenization is a type of subword tokenization. Subword tokenization is a technique for splitting words into smaller units, called subwords, that are still meaningful. This is in contrast to traditional word tokenization, which simply splits words on whitespace or punctuation.

WordPiece tokenization works by first creating a vocabulary of subwords. This vocabulary is created by iteratively merging the most common subwords together until the desired vocabulary size is reached. The merging process is done in a way that minimizes the information loss.

When a word is tokenized using WordPiece, it is first pre-tokenized into words by splitting on whitespace and punctuation. Then, each word is tokenized into subwords using the vocabulary. If a word is not in the vocabulary, it is split into subwords using the merging process.

WordPiece tokenization is effective for handling rare/OOV words because it allows the model to learn the meaning of words even if they are not in the vocabulary. This is because the model can learn the meaning of a word from the subwords that make up the word.

For example, the word "playing" is not in the vocabulary of a WordPiece model. However, the model can learn the meaning of the word by learning the meaning of the subwords "play" and "ing".

WordPiece tokenization is a powerful technique for handling rare/OOV words in NLP models. It is used in many popular models, such as BERT, DistilBERT, and Electra.