What are the advantages of using different tokenizers? For example, let's take the sentence: "In Düsseldorf I took my hat off. But I can't put it back on."

The treebank tokenizer yields: "In Düsseldorf I took my hat off . But I ca n't put it back on . "

However, the whitespace tokenizer would yield: "In Düsseldorf I took my hat off . But I can't put it back on . "

NLTK has four tokenizers:

  • TreebankWordTokenizer
  • WordPunctTokenizer
  • PunctWordTokenizer
  • WhitespaceTokenizer

When should you use which one? For my project, I am interested in text generation, so I am leaning toward the whitespace tokenizer. Is this a good choice? Won't my model generate nonsense tokens like "n't" when I use eg the treebank tokenizer?

  • $\begingroup$ comment as an answer due to insufficient karma. I would suggest trying all four and seeing how the results look. $\endgroup$
    – user70889
    Oct 5 at 13:12

1 Answer 1


The problem is of text generation. I am assuming you are trying for chatbot etc where input is a natural lanugae and output is a natural language.

Since input is a natural language, all punctuations,special characters are important. For eg: Triple dot also means " to follow up" or "waiting". A tokenizer based on "." will remove this information.

Next step is to choose tokenizer which preserves punctuations. Tokenizer based on white space will do.

  • $\begingroup$ Thank you for tha answer and your reasoning. I was also asking because a previous paper author used the treebank tokenizer to preprocess their training data which confused me. Creating a vocabulary based on nonsense tokens like "n't" seemed problematic to me. This reinforces my hunch that a whitespace tokenizer is the better choice indeed. $\endgroup$ Oct 6 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.