The title says it all. I was researching this question but couldn't find something useful. What is the difference between adding words to a tokenizer and training a tokenizer?
1 Answer
$\begingroup$
$\endgroup$
1
First, a clarification: tokenizers receive text and return tokens. These tokens may be words or not. Some tokenizers, for instance, return word pieces (i.e. subwords). This way, a single word may lead to multiple tokens (e.g. "magnificently" --> ["magn", "ific", "ently"]). Some examples of subword tokenizers are Byte-Pair Encoding (BPE) and Unigram. Therefore, adding a "word" to a tokenizer may not make sense for a subword-level tokenizer; instead, I will refer to it as "adding a token".
Some simple tokenizers rely on pre-existing boundaries between tokens. For instance, it is very common to tokenize by relying on the white space between words (after a previous mild pre-processing to separate punctuation).
Depending on the complexity of the separation of tokens from the text, the tokenization process can consist of just a lookup in a table to a complex computation of probabilities.
For simple tokenizers that only consist of a lookup table, adding a token to it is simple: you just add an entry to the table.
For more complex tokenizers, you need a training process that learns the needed information to later tokenize. In those cases, adding a token is simply not possible, because the information stored in the tokenizer is richer, not just a table with entries.
