0
$\begingroup$

What are the advantages of using different tokenizers? For example, let's take the sentence: "In Düsseldorf I took my hat off. But I can't put it back on."

The treebank tokenizer yields: "In Düsseldorf I took my hat off . But I ca n't put it back on . "

However, the whitespace tokenizer would yield: "In Düsseldorf I took my hat off . But I can't put it back on . "

NLTK has four tokenizers:

  • TreebankWordTokenizer
  • WordPunctTokenizer
  • PunctWordTokenizer
  • WhitespaceTokenizer

When should you use which one? For my project, I am interested in text generation, so I am leaning toward the whitespace tokenizer. Is this a good choice? Won't my model generate nonsense tokens like "n't" when I use eg the treebank tokenizer?

Improve this question
$\endgroup$
1
  • $\begingroup$ comment as an answer due to insufficient karma. I would suggest trying all four and seeing how the results look. $\endgroup$
    – user70889
    Oct 5, 2022 at 13:12

1 Answer 1

Reset to default
2
$\begingroup$

The problem is of text generation. I am assuming you are trying for chatbot etc where input is a natural lanugae and output is a natural language.

Since input is a natural language, all punctuations,special characters are important. For eg: Triple dot also means " to follow up" or "waiting". A tokenizer based on "." will remove this information.

Next step is to choose tokenizer which preserves punctuations. Tokenizer based on white space will do.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for tha answer and your reasoning. I was also asking because a previous paper author used the treebank tokenizer to preprocess their training data which confused me. Creating a vocabulary based on nonsense tokens like "n't" seemed problematic to me. This reinforces my hunch that a whitespace tokenizer is the better choice indeed. $\endgroup$
    – postnubilaphoebus
    Oct 6, 2022 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.