When i use T5TokenizerFast(Tokenizer of T5 arcitecture), the output is expected as follows:

['▁', '</s>', '▁Hello', '▁', '<sep>', '</s>']

But when i use normal tokenizer, it starts to split special token "/s>" as follows:

['▁</', 's', '>', '▁Hello', '<sep>', '</s>']

And this is print of not fast tokenizer:

PreTrainedTokenizer(name_or_path='', vocab_size=60000, model_max_len=1000000000000000019884624838656, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'eos_token': '</s>', 'unk_token': '<unk>', 'pad_token': '<pad>'})

For fast :

PreTrainedTokenizerFast(name_or_path='', vocab_size=60000, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'eos_token': '</s>', 'unk_token': '<unk>', 'pad_token': '<pad>'})

Code that i am using to produce these outputs:

tokenizer = T5TokenizerFast('new_sp.model', extra_ids=0)
print(tokenizer.convert_ids_to_tokens(tokenizer.encode("</s> Hello <sep>")))

I would appreciate any help. Thanks.

  • 1
    $\begingroup$ Maybe this is a bug in the huggingface transformers library. Have you checked if there is a similar issue on their github? I see there are a number of issues regarding discrepancies between the fast and not fast tokenizers (see this). $\endgroup$
    – noe
    Jul 30 at 15:30
  • $\begingroup$ @noe Yes, i am trying them right now. When i got the results i will update my question. Thanks noe :) $\endgroup$
    – canP
    Jul 30 at 16:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.