The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and provides a method to perform sentence segmentation on unseen corpus.

NLTK provides word_tokenize and sent_tokenize. Creating our own tokenizers can help us understand how one works, but in a production environment why would we want a custom tokenizer? And if I built a custom tokenizer, how could I measure if it was better that NLTK's tokenizer?

  • $\begingroup$ Which chapter in the book talks about sentence segmentation using Naive Bayes? $\endgroup$ Commented Aug 29, 2018 at 5:55

1 Answer 1


Why would we want a custom tokenizer?

Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer. Any toolkit needs to be flexible, and the ability to change the tokenizer, both so that someone can experiment, and so that it can be replaced if requirements are different, or better ways are found for specific problems, is useful and important.

How could I measure if it was better that NLTK's tokenizer?

Anytime you are trying to quantify performance (ie: better) you will need to first define what is meant by better. Once this is done, then typically you would perform the using the various methods under measurement, and then compare the results against your definition of better. A couple of links which discuss these topics:

  • $\begingroup$ hello, I've noticed that you've not come here for days. I guess you are a bit angry due to our debate. If I've make you upset, please accept my apologies. I searched your name and saw your image in your github page. You are very older than me, it was wrong to argue with you. Maybe I have the age of your childeren and I'm not treated in a way not to respect someone which is older than me. If I've make you upset, I'm sorry for that. Don't leave here. We all are friends. Thanks you sir :) $\endgroup$ Commented Dec 28, 2018 at 19:32
  • $\begingroup$ @Media, I am not angry with you. I am disappointed in the moderators who do not want to close off-topic questions. I have spent many many hours doing over 6000 reviews, 1200 edits and 500 flags in an effort to achieve the stated objective of a high quality Question and Answer site. But in the end the moderators set the tone for the site, and if they are more concerned with how you feel, then whether the question is off topic, then I was wasting my time. Thank you for the best wishes, and good luck to you. $\endgroup$
    – Stephen Rauch
    Commented Dec 28, 2018 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.