How was the token library constructed for ChatGPT / other GPT systems?

Question

I have found literally hundreds of articles on Google with titles like 'What are tokens and how to use them,' but haven't been able to find any information at all on how the token libraries themselves were made. OpenAI state that GPT-3.5 uses a token library of 50,257 tokens, which presumably can represent any string in their training library. But how were these tokens selected? I assume there was some kind of optimization algorithm that found the most common strings in the training library? Was the tokenizer trained alongside the GPT models to optimize both for one another? There are a lot of strange cases, particularly around how spaces are handled (see below for a few examples I made of closely related words represented by different tokens), so I'm really curious what the methodology of making a token set is. Obviously how tokens are made will have a huge impact on how the system as a whole performs, yet unlike the attention mechanism very little seems to have been written about it.

Answer 1

ChatGPT uses byte-pair encoding (BPE) as tokenization strategy. This approach was first proposed in the scientific article Neural Machine Translation of Rare Words with Subword Units. BPE uses subword-level tokens. To define the list of subwords, the BPE algorithm uses a text corpus and tries to find the most reusable word parts. Also, the BPE vocabulary keeps the individual characters as tokens, which grants it the ability to represent any word that is written in the same scripts it was trained on.

The canonical implementation of BPE is this Python package created and maintained by the first author of the scientific article. However, OpenAI created their own implementation, called Tiktoken, which is faster.

Other popular tokenization strategies include "wordpieces" (which is very similar to BPE)(see this answer for details) and the unigram vocabulary (see this answer for details).