I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text tokenizer. I have never seen a subword tokenizer before and don't know why or when it should be used as opposed to a word tokenizer.

The tutorial says The tokenizer encodes the string by breaking it into subwords if the word is not in its dictionary.

To get an idea of what the results can look like, the work Transformer gets broken down into index-subword pairs.

7915 ----> T
1248 ----> ran
7946 ----> s
7194 ----> former

Does anybody know what the advantages of breaking down words into subwords is and when somebody should use a subword tokenizer instead of the more standard word tokenizer? Is the subword tokenizer used because the translation is from Portuguese to English?

*The version of Tensorflow is 2.3 and this subword tokenizer belongs to tfds.deprecated.text


1 Answer 1


Subword tokenization is the norm nowadays in NLP models because:

  • It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. This is a problem for morphologically-rich languages, proper nouns, etc. Subword vocabularies allow representing these words. By having subword tokens (and ensuring the individual characters are part of the subword vocabulary), makes it possible to encode words that were not even in the training data. There's still the problem with characters not present in the training data, but that's tolerable in most of the cases.

  • It gives manageable vocabulary sizes. Current neural networks need a pre-defined closed discrete token vocabulary. The vocabulary size that a neural network can handle is far smaller than the number of different words (surface forms) in most normal languages, especially morphologically-rich ones (and especially agglutinative ones).

  • Mitigates data sparsity. In a word-based vocabulary, low-frequency words may appear very few times in the training data. This is especially troublesome for agglutinative languages, where a surface form may be the result of concatenating multiple affixes. Using subword tokenization allows token reusing, and increases the frequency of their appearance.

  • Neural networks perform very well with them. In all sorts of tasks, they excel: neural machine translation, NER, etc, you name it, the state of the art models are subword-based: BERT, GPT-3, Electra,...

  • $\begingroup$ I see, nice explanation. So, for your second point about manageable vocab sizes, the subwords don't cause any problems given that they make a larger vocab size compared to the older word tokenization, right? $\endgroup$
    – zipline86
    Commented Oct 9, 2020 at 11:23
  • $\begingroup$ Yes, having a 30k subword vocabulary for a single language is usually enough to perform appropriately, while a 30k word vocabulary is very very small for any morphologically-rich language. $\endgroup$
    – noe
    Commented Oct 9, 2020 at 11:42
  • $\begingroup$ Last question and thanks for all the helpful answers. Do you if there's a standard go to vocabulary size for a single language that people usually use or is it one of those things that have to be played around with to see what gives the best results? $\endgroup$
    – zipline86
    Commented Oct 9, 2020 at 11:48
  • $\begingroup$ If you have plenty of data, the typical number is 32-40k for one language. The same range is used for a combined vocabulary in machine translation when both languages share the same script. If you have little data, people tend to use 10-15k. You can check this article which studies the influence of the number of subwords in NMT systems. $\endgroup$
    – noe
    Commented Oct 9, 2020 at 12:01
  • 1
    $\begingroup$ If you have very little data, going even lower than 10k (e.g. 2k) can make quite a difference in my personal experience. See for instance: aclweb.org/anthology/P19-1021.pdf. $\endgroup$ Commented Oct 9, 2020 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.