I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-Vocabulary (OOV) words. Can someone please help me explain how WordPiece tokenization is actually done, and how it handles effectively helps to rare/OOV words?

  • 3
    $\begingroup$ This question has been answered here. I'm copying the answer here as well. WordPiece and BPE are two similar and commonly used techniques to segment words into subword-level in NLP tasks. In both cases, the vocabulary is initialized with all the individual characters in the language, and then the most frequent/likely combinations of the symbols in the vocabulary are iteratively added to the vocabulary. Consider the WordPiece algorithm from the [original paper](static.googleusercontent.com/media/research.google.com/en//pubs/… $\endgroup$
    – Harman
    Apr 2, 2019 at 12:35
  • 2
    $\begingroup$ The URL is broken. Here the corrected one. $\endgroup$
    – Laura
    Aug 22, 2019 at 14:52
  • $\begingroup$ Here you go... youtube.com/watch?v=zJW57aCBCTk | I've recently started following this guy. He has a very nice way of explaining such concepts. $\endgroup$
    – Aditya
    Jan 9, 2020 at 7:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.