I have been reading about both these techniques to find the root of the word, but how do we prefer one to the other?
Is "Lemmatization" always better than "Stemming"?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
I would say that lemmatization is generally the preferred way of reducing related words to a common base.
This Quora question is a good resource on the subject: Is it advisable to choose lemmatization over stemming in NLP? The top answer quotes another good resource that motivates why lemmatization is usually better, Stemming and lemmatization, from Stanford NLP:
Why lemmatization is better
Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes.
Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.
But that is just generally, it is not always better. Stemming still has some advantages and it will depend on the use case. Some reasons you would use stemming over lemmatization could be:
Some possible exceptions when stemming can be better
- Simplicity
- Speed
- Memory constraints
answered Apr 22, 2019 at 11:19
-
4$\begingroup$ Another case where I personally found stemming to (sometimes) work better: very short text snippets like business names, which often don't fully obey grammar rules; aggressive stemming algorithms like Snowball tend to work just as well if not better in my experience. $\endgroup$ Commented Apr 22, 2019 at 17:04
-
$\begingroup$ Thanks for adding! I will edit to make it more clear that there can be more situations where stemming can work better. $\endgroup$ Commented Apr 22, 2019 at 17:22
-
4$\begingroup$ I can't emphasize enough on the speed! I had an app which took forever to run because we were running lemmatization. Replacing it with stemming accelerated things up. $\endgroup$ Commented Apr 22, 2019 at 17:28
$\begingroup$
$\endgroup$
It really depends on your use case. Whenever we talk about projects where speed and complexity do not play a role lemmatization is usually the better solution. It is also the better solution since it outputs actual words and not just word stems which do not have to be real words.
Stemming is just preferred when you have to process a lot of words and are restricted in processing power.
However, nowadays transformer models detect the meaning and context the words are used in pretty well. Then you don't need to preprocess the data with stemming or lemmatization.
