I have been reading about both these techniques to find the root of the word, but how do we prefer one to the other?
Is "Lemmatization" always better than "Stemming"?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
I would say that lemmatization is generally the preferred way of reducing related words to a common base.
This Quora question is a good resource on the subject: Is it advisable to choose lemmatization over stemming in NLP? The top answer quotes another good resource that motivates why lemmatization is usually better, Stemming and lemmatization, from Stanford NLP:
Why lemmatization is better
Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes.
Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.
But that is just generally, it is not always better. Stemming still has some advantages and it will depend on the use case. Some reasons you would use stemming over lemmatization could be:
Some possible exceptions when stemming can be better
It really depends on your use case. Whenever we talk about projects where speed and complexity do not play a role lemmatization is usually the better solution. It is also the better solution since it outputs actual words and not just word stems which do not have to be real words.
Stemming is just preferred when you have to process a lot of words and are restricted in processing power.
However, nowadays transformer models detect the meaning and context the words are used in pretty well. Then you don't need to preprocess the data with stemming or lemmatization.