The title says it all. I was researching this question but couldn't find something useful. What is the difference between adding words to a tokenizer and training a tokenizer?


1 Answer 1


First, a clarification: tokenizers receive text and return tokens. These tokens may be words or not. Some tokenizers, for instance, return word pieces (i.e. subwords). This way, a single word may lead to multiple tokens (e.g. "magnificently" --> ["magn", "ific", "ently"]). Some examples of subword tokenizers are Byte-Pair Encoding (BPE) and Unigram. Therefore, adding a "word" to a tokenizer may not make sense for a subword-level tokenizer; instead, I will refer to it as "adding a token".

Some simple tokenizers rely on pre-existing boundaries between tokens. For instance, it is very common to tokenize by relying on the white space between words (after a previous mild pre-processing to separate punctuation).

Depending on the complexity of the separation of tokens from the text, the tokenization process can consist of just a lookup in a table to a complex computation of probabilities.

For simple tokenizers that only consist of a lookup table, adding a token to it is simple: you just add an entry to the table.

For more complex tokenizers, you need a training process that learns the needed information to later tokenize. In those cases, adding a token is simply not possible, because the information stored in the tokenizer is richer, not just a table with entries.

  • $\begingroup$ Thank you so much for your clear explanation. $\endgroup$
    – canP
    Jul 23, 2022 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.