What are the advantages of using different tokenizers? For example, let's take the sentence: "In Düsseldorf I took my hat off. But I can't put it back on."
The treebank tokenizer yields: "In Düsseldorf I took my hat off . But I ca n't put it back on . "
However, the whitespace tokenizer would yield: "In Düsseldorf I took my hat off . But I can't put it back on . "
NLTK has four tokenizers:
- TreebankWordTokenizer
- WordPunctTokenizer
- PunctWordTokenizer
- WhitespaceTokenizer
When should you use which one? For my project, I am interested in text generation, so I am leaning toward the whitespace tokenizer. Is this a good choice? Won't my model generate nonsense tokens like "n't" when I use eg the treebank tokenizer?