The Natural Language Processing with Python book is a really good resource to understand basics of NLP. One of the chapters introduces training 'sentence segmentation' using Naive Bayes Classifer and provides a method to perform sentence segmentation on unseen corpus.
NLTK provides word_tokenize
and sent_tokenize
. Creating our own tokenizers can help us understand how one works, but in a production environment why would we want a custom tokenizer? And if I built a custom tokenizer, how could I measure if it was better that NLTK's tokenizer?