1
vote
1
vote
dealing with HuggingFace's model's tokens
For your first question, you can check if the tokenizer covers a certain string with the following:
...
Only top scored, non community-wiki answers of a minimum length are eligible
Related Tags
bart × 5huggingface × 3
bert × 2
nlp × 1
pandas × 1
pytorch × 1
transformer × 1
matplotlib × 1
plotting × 1
machine-translation × 1
text-generation × 1
tokenization × 1
automatic-summarization × 1
pretraining × 1