I'm building an application for the API, but I would like to be able to count the number of tokens my prompt will use, before I submit an API call. Currently I often submit prompts that yield a 'too-many-tokens' error.

The closest I got to an answer was this post, which still doesn't say what tokenizer it uses.

If I knew what tokenizer the API used, then I could count how many tokens are in my prompt before I submit the API call.

I'm working in Python.


1 Answer 1


Tokenizer for GPT-3 is the same as GPT-2:


linked via:


UPDATE March 2023

For newer models, including GPT-3.5 (turbo), GPT-4, and latest embeddings, use tiktoken tokenizer with the cl100k_base encoding:


A full model-to-encoding mapping can be found here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.