I'm looking at this Tensorflow colab tutorial about language translation with Transformers, https://www.tensorflow.org/tutorials/text/transformer, and they tokenize the words with a subword text tokenizer. I have never seen a subword tokenizer before and don't know why or when it should be used as opposed to a word tokenizer.
The tutorial says
The tokenizer encodes the string by breaking it into subwords if the word is not in its dictionary.
To get an idea of what the results can look like, the work
Transformer gets broken down into index-subword pairs.
7915 ----> T 1248 ----> ran 7946 ----> s 7194 ----> former
Does anybody know what the advantages of breaking down words into subwords is and when somebody should use a subword tokenizer instead of the more standard word tokenizer? Is the subword tokenizer used because the translation is from Portuguese to English?
*The version of Tensorflow is 2.3 and this subword tokenizer belongs to tfds.deprecated.text