2
$\begingroup$

For text processing there are plenty of tools out there like CoreNLP, SpaCy, NLTK, textblob etc. each offering different suites of pre-processing functions and people recommend different tool for different tasks like NLTK for tokenization etc.

Which tool would you recommend which performs all these tasks?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Comparing two libraries or tools in terms of these things is somewhat that is opinion dependent. Some people prefer NLTK for doing almost all the tasks. spacy has also gained quite reputation. But what is better for you depends on what you want do. In my personal experience, I have found that NLTK along with gensim libraries is all that I need to do all the Natural Language Processing tasks.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Yeah, makes sense. I am guessing you use NLTK for tokenization? Which tokenizer (general purpose) you recommend using from the tokenizers NLTK offers? My main source of confusion is availability of multiple tokenizers by each library. $\endgroup$
    – SHASHANK GUPTA
    Commented Nov 26, 2016 at 17:18
  • $\begingroup$ Yea NLTK for tokenization and other preprocessing like removal of stopwords,etc. Also, you can use BeautifulSoup if your text contains html. $\endgroup$
    – enterML
    Commented Nov 26, 2016 at 17:57
  • $\begingroup$ Sure, thanks. BTW which tokenizer do you use generally in NLTK? I read its documentation and there are multiple tokenizers available. $\endgroup$
    – SHASHANK GUPTA
    Commented Nov 26, 2016 at 17:59
  • $\begingroup$ punkt tokenizer in general. And if you have find this answer useful, please accept it. $\endgroup$
    – enterML
    Commented Nov 26, 2016 at 18:00
  • $\begingroup$ Can you compare nltk's tokenizer with nltk punkt tokenizer? Punkt tokenizer includes punctuations also in the tokens, whereas gensim's tokenizer gives only work tokens without any punctuation. Is that a bad thing, to have units of two word tokens in your vocab too? $\endgroup$
    – SHASHANK GUPTA
    Commented Nov 29, 2016 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.