7
$\begingroup$

I have been reading about both these techniques to find the root of the word, but how do we prefer one to the other?

Is "Lemmatization" always better than "Stemming"?

$\endgroup$
8
$\begingroup$

I would say that lemmatization is generally the preferred way of reducing related words to a common base.

This Quora question is a good resource on the subject: Is it advisable to choose lemmatization over stemming in NLP? The top answer quotes another good resource that motivates why lemmatization is usually better, Stemming and lemmatization, from Stanford NLP:

Why lemmatization is better

Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes.

Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.

But that is just generally, it is not always better. Stemming still has some advantages and it will depend on the use case. Some reasons you would use stemming over lemmatization could be:

Some possible exceptions when stemming can be better

  • Simplicity
  • Speed
  • Memory constraints
$\endgroup$
  • 3
    $\begingroup$ Another case where I personally found stemming to (sometimes) work better: very short text snippets like business names, which often don't fully obey grammar rules; aggressive stemming algorithms like Snowball tend to work just as well if not better in my experience. $\endgroup$ – shadowtalker Apr 22 at 17:04
  • $\begingroup$ Thanks for adding! I will edit to make it more clear that there can be more situations where stemming can work better. $\endgroup$ – Simon Larsson Apr 22 at 17:22
  • 3
    $\begingroup$ I can't emphasize enough on the speed! I had an app which took forever to run because we were running lemmatization. Replacing it with stemming accelerated things up. $\endgroup$ – ChiPlusPlus Apr 22 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.