I'm building an application for the API, but I would like to be able to count the number of tokens my prompt will use, before I submit an API call. Currently I often submit prompts that yield a 'too-many-tokens' error.
The closest I got to an answer was this post, which still doesn't say what tokenizer it uses.
If I knew what tokenizer the API used, then I could count how many tokens are in my prompt before I submit the API call.
I'm working in Python.