I have raised a ticket in github, whose link is here
Which has a problem statement as mentioned below. If you have any solution to this, do answer. I am trying to make a language model using a transformer from scratch, For that, I want to build a tokenizer that tokenizes a text data using whitespace only, nothing else. It should generate a vocab file that does not have any special character, just the words separated by whitespace and then I want to use that tokenizer to build a language model from scratch using this.
I don't want my tokenizer to generate vocabs that have any kind of special characters viz "##" in front of words and any accents in my vocab.
I know there are tokenizers that give good results for language models like bpe and workpiece, but I have a requirement where I just want to use whitespace tokenizer only for training a language model.