I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-Vocabulary (OOV) words. Can someone please help me explain how WordPiece tokenization is actually done, and how it handles effectively helps to rare/OOV words?
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ This question has been answered here. I'm copying the answer here as well. WordPiece and BPE are two similar and commonly used techniques to segment words into subword-level in NLP tasks. In both cases, the vocabulary is initialized with all the individual characters in the language, and then the most frequent/likely combinations of the symbols in the vocabulary are iteratively added to the vocabulary. Consider the WordPiece algorithm from the [original paper](static.googleusercontent.com/media/research.google.com/en//pubs/… $\endgroup$– HarmanApr 2, 2019 at 12:35
-
2$\begingroup$ The URL is broken. Here the corrected one. $\endgroup$– LauraAug 22, 2019 at 14:52
-
$\begingroup$ Here you go... youtube.com/watch?v=zJW57aCBCTk | I've recently started following this guy. He has a very nice way of explaining such concepts. $\endgroup$– AdityaJan 9, 2020 at 7:54
Add a comment
|