1
$\begingroup$

I am using Hunspell to spellcheck and stem the words in my documents to reduct dimensionality. For spellchecking Hunspell works great with the default en_US dictionary by SCOWL (and friends), but not so for stemming.

The reason is that the dictionary is pretty inconsistent. (e.g. some words have their plurals in the dictionary, but some others doesn't, so it's inconsistent if the stem of a word in plural will be in plural or not).

Hunspell is definitely able to stem english words consistently, but it needs an other dictionary that is more "correct" morphologically.

  1. What alternative dictionaries can I use to improve the quality of stemming English text with Hunspell?

  2. If there are no such dictionaries, what other tools can I use for stemming? EDIT: note that I'm using python, which can be hooked up with most libraries and command-line tools easily, but R or Matlab specific tools might not fit in my current ecosystem that well.

Improve this question
$\endgroup$
2
  • $\begingroup$ The most popular text mining package in python is called ntlk. It has a stemming module. Is it not good enough for what you want? $\endgroup$
    – Ricardo Magalhães Cruz
    Aug 23, 2016 at 16:44
  • $\begingroup$ The best solution would be a configurable, extendable, dictionary-based stemming algorithm that produces meaningful words as stems, while keeping the part of speech. Hunspell is such, for many languages, except for English, since I couldn't find the right dictionary. My second choice would be the Porter stemmer from nltk though, due to it's simplicity. $\endgroup$
    – mimrock
    Aug 23, 2016 at 20:38

1 Answer 1

Reset to default
3
$\begingroup$

I can't comment on your question (my reputation isn't high enough), but I've got a few clarifying questions to ask: Why do you want to stem words? Text classification? Sentiment analysis? Something else?

I've used both the 'tm' package and the 'RTextTools' package for stemming when doing text classification. Both have some built-in functions for stemming.

Whether they would be useful to you or not depends on why you want to stem words ...

FWIW, here are some links related to those packages:

Improve this answer
$\endgroup$
4
  • $\begingroup$ I stem the words to reduct dimensionality. I use gold standard corpus, so my training set is not that large. The current goal is some kind of text classification. With less dimensions I expect to get better results with the same amount of training data so that's why I think I need to correct mispelled words, and remove affixes. $\endgroup$
    – mimrock
    Aug 23, 2016 at 14:52
  • $\begingroup$ The 'tm' package allows the creation of a corpus with ability to stem words in the corpus. This will definitely reduce the dimensions in any document-term-matrix or term-document-matrix you might want to create. However, it relies on an algorithm to stem, as opposed to a dictionary. See the tm::stemDocument function for more info. I believe RTextTools does similarly, but without direct corpus creation. $\endgroup$
    – jab
    Aug 23, 2016 at 16:08
  • $\begingroup$ Sorry, just saw your edit that you are working in Python. Those two packages are for R. $\endgroup$
    – jab
    Aug 24, 2016 at 12:43
  • $\begingroup$ It's not you. I added that after I've seen your answer. Anyway, thanks for sharing your ideas. $\endgroup$
    – mimrock
    Aug 25, 2016 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.