1
vote
1
vote
dealing with HuggingFace's model's tokens
For your first question, you can check if the tokenizer covers a certain string with the following:
...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible