I have raised a ticket in github, whose link is here

Which has a problem statement as mentioned below. If you have any solution to this, do answer. I am trying to make a language model using a transformer from scratch, For that, I want to build a tokenizer that tokenizes a text data using whitespace only, nothing else. It should generate a vocab file that does not have any special character, just the words separated by whitespace and then I want to use that tokenizer to build a language model from scratch using this.

I don't want my tokenizer to generate vocabs that have any kind of special characters viz "##" in front of words and any accents in my vocab.

I know there are tokenizers that give good results for language models like bpe and workpiece, but I have a requirement where I just want to use whitespace tokenizer only for training a language model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.