I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-Vocabulary (OOV) words. Can someone please help me explain how WordPiece tokenization is actually done, and how it handles effectively helps to rare/OOV words?

  • 2
    $\begingroup$ This question has been answered here. I'm copying the answer here as well. WordPiece and BPE are two similar and commonly used techniques to segment words into subword-level in NLP tasks. In both cases, the vocabulary is initialized with all the individual characters in the language, and then the most frequent/likely combinations of the symbols in the vocabulary are iteratively added to the vocabulary. Consider the WordPiece algorithm from the [original paper](static.googleusercontent.com/media/research.google.com/en//pubs/… $\endgroup$ – Harman Apr 2 '19 at 12:35
  • 2
    $\begingroup$ The URL is broken. Here the corrected one. $\endgroup$ – Laura Aug 22 '19 at 14:52
  • $\begingroup$ Here you go... youtube.com/watch?v=zJW57aCBCTk | I've recently started following this guy. He has a very nice way of explaining such concepts. $\endgroup$ – Aditya Jan 9 '20 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.