From what I understand, GPT and GPT-2 are trained to predict the $N^{th}$ word in a sentence given the previous $N-1$ words. When the vocabulary size is very large (100k+ words) how is it able to generate any meaningful prediction? Shouldn't it become extremely difficult to correctly label the next word given that there are 100k possible labels to choose from? Even a large-scale classification problem like ImageNet has only 1k classes to choose from.


1 Answer 1


GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token.

BPE tries to find the pieces of words that are most reusable. BPE also keeps character subwords (e.g. "a", "W").

The subword vocabulary used by GPT-2 is 50k tokens big. You can take a look at the vocabulary here. There, you can see that token #12175 is "retro" and token #11147 is "fit", so when tokenizing word "retrofit", you would probably get the 2 subword tokens #68 and #12541. When tokenizing a low-frequency word (i.e. that did not appear in the BPE training data), you probably end up with small subwords tokenization, e.g. "Kilimanjaro" --> "Kil", "iman", "jar", "o".

Using a word-level vocabulary is very troublesome, because:

  1. the amount of existing word surface forms exceeds the size that it is manageable for a neural net, even for a morphologically simple language like English. For morphologically-rich fusional languages and, especially, agglutinative languages, using word-level vocabularies is much more inconvenient.
  2. language is "open", in the sense that new words can be created, either by stitching together existing words (e.g. manspreading), or by making up new words altogether (e.g. bazinga).
  3. the less frequently a token appears in the training data, the less our net will learn to use it. If we have many many different tokens, the frequency of appearance of each token in the training data will be very low (i.e. data scarcity), so our net will not learn properly.

Therefore, subword vocabularies are the norm nowadays.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.