The typical default for neural networks in natural language processing has been to take words as tokens.

OpenAI Codex is based on GPT-3, but also deals with source code. For source code in general, there is no corresponding obvious choice of tokens, because each programming language has different rules for tokenizing. I don't get the impression Codex uses a separate tokenizer for each language.

What does it take as tokens?


1 Answer 1


NLP neural networks don't use word tokens any more. It's been a while since the norm is using subwords. Usual approaches to define the subword vocabulary are byte-pair encoding (BPE), word pieces or unigram tokenization.

GPT-3 uses BPE tokenization. According to the OpenAI's tokenizer tool website:

Codex models use a different set of encodings that handle whitespace more efficiently

From this, I understand that they use BPE but with a different vocabulary. This is supported by this javascript tokenizer that was created by extracting the BPE vocabulary from OpenAI's own online tokenizer tool.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.