1
$\begingroup$

For text processing there are plenty of tools out there like CoreNLP, SpaCy, NLTK, textblob etc. each offering different suites of pre-processing functions and people recommend different tool for different tasks like NLTK for tokenization etc.

Which tool would you recommend which performs all these tasks?

$\endgroup$
3
$\begingroup$

Comparing two libraries or tools in terms of these things is somewhat that is opinion dependent. Some people prefer NLTK for doing almost all the tasks. spacy has also gained quite reputation. But what is better for you depends on what you want do. In my personal experience, I have found that NLTK along with gensim libraries is all that I need to do all the Natural Language Processing tasks.

$\endgroup$
  • $\begingroup$ Yeah, makes sense. I am guessing you use NLTK for tokenization? Which tokenizer (general purpose) you recommend using from the tokenizers NLTK offers? My main source of confusion is availability of multiple tokenizers by each library. $\endgroup$ – SHASHANK GUPTA Nov 26 '16 at 17:18
  • $\begingroup$ Yea NLTK for tokenization and other preprocessing like removal of stopwords,etc. Also, you can use BeautifulSoup if your text contains html. $\endgroup$ – Nain Nov 26 '16 at 17:57
  • $\begingroup$ Sure, thanks. BTW which tokenizer do you use generally in NLTK? I read its documentation and there are multiple tokenizers available. $\endgroup$ – SHASHANK GUPTA Nov 26 '16 at 17:59
  • $\begingroup$ punkt tokenizer in general. And if you have find this answer useful, please accept it. $\endgroup$ – Nain Nov 26 '16 at 18:00
  • $\begingroup$ Can you compare nltk's tokenizer with nltk punkt tokenizer? Punkt tokenizer includes punctuations also in the tokens, whereas gensim's tokenizer gives only work tokens without any punctuation. Is that a bad thing, to have units of two word tokens in your vocab too? $\endgroup$ – SHASHANK GUPTA Nov 29 '16 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.