Hi im just getting started with understanding transformer based models and I am not able to find how the token embeddings are arrived at?. there are multiple tokenization approaches and multiple vocabularies/documents llms are trained on. so my question is

  1. whether each llm also trains its own token embeddings?
  2. how do those pre trained embeddings work for transfer learning or fine tuning, on custom data sets where some OOV words may be present or we have some special unique tokens we want to keep whole and not have tokenizer do subword tokens?

2 Answers 2


First, the token vocabulary is extracted from the training data, usually by means of byte-pair encoding (BPE), wordpieces or unigrams. Then, regarding model definition, the first layer in LLMs is the token embeddings, which are trained with the rest of the model.

Given that subword-level vocabularies mitigate the OOV problem, there are no unknown tokens, so usually there is no need to address such a problem. It is not common either to have special tokens that need to be kept unsplit.

In general, it is inconvenient to modify the vocabulary of a pretrained model, because you need to retrain/finetune the model to make it able to handle the new tokens, and therefore you lose the advantages of reusing the pretrained representations; furthermore, the fine-tuning data may not be enough to deliver good representations for new tokens.

  • $\begingroup$ couple of questions? 1. how do subword level vocab mitigate OOV problems? do they tokenize by each byte? or all unicode character representations?. I mean thats the only way to ensure no unicode based text wont have oov 2. even if they do they may not be optimal for my use case. a lot of the token embeddings contain information about the pre training corpus. and splitting up special tokens for my use case will not give me the same results as when it would be whole. $\endgroup$
    – dasman
    Commented Aug 24, 2023 at 7:01
  • $\begingroup$ Subword-level vocabs contain tokens of different granularities: whole words, subwords, characters and bytes. They will not suffer from the OOV because they can represent anything. Of course, words in scripts less present in the training data will need more tokens to be represented than words in Latin script. Likewise, the LLM will be less able to handle such under-represented scripts. If you are dealing with such a case, a common approach is to try and find a model that has been trained on data where your target language is more represented. $\endgroup$
    – noe
    Commented Aug 24, 2023 at 11:01
  1. Yes, each model has a different tokenizer type. You can see the list on the different types of tokenizers in the documentation for AutoTokenizer
  2. You can use the same type of tokenizer as the 'flavor' of model your transfer learning is based upon and train your own tokenizer. See the video here: Training a new tokenizer from an old one.

Regarding OOV tokens, there is also utility method that might be worth looking into, add_token, though it appears there are some nuances to be considered there.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.