2
$\begingroup$
Who teaches English?

Now, after tokenizing, stemming.. it gives me

Who, teach, English

In my list of word, I have a word called

teacher

Lemmatizing, stemming teacher gives teacher and lemmatizing, stemming teaches gives teach

Even, calculating edit_distance will not solve this.. As, edit_distance comes out to be 2.

Now, What do I do to have teacher and teach treated as similar? Similarly, there may be other cases with extra 's' at the end. Is there some stemmer that solves this problem? Is there any solution?

Other similar example can be: instructor and instructs

$\endgroup$
1
$\begingroup$

Use an aggressive stemmer. The Lancaster Stemmer is one the most aggressive and popular stemmers around.

Here is the Python code:

from nltk.stem.lancaster import LancasterStemmer

lancaster_stemmer = LancasterStemmer()
assert 'teach' == lancaster_stemmer.stem('teacher') == lancaster_stemmer.stem('teaches')
| improve this answer | |
$\endgroup$
1
$\begingroup$

Check out Fasttext. Fasttext works similarly to word2vec in that you can create word embeddings, however, it actually analyzes character n-grams, to force the syntactic similarity to what you're thinking about.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I tried using fasttext. But, found some problem. @j.a.gartner, Could you please see this query $\endgroup$ – Sabbiu Shah Jun 30 '17 at 2:17
  • $\begingroup$ It is now solved! $\endgroup$ – Sabbiu Shah Jun 30 '17 at 11:16
  • $\begingroup$ Sorry, didn't see until you had already solved it. $\endgroup$ – j.a.gartner Jun 30 '17 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.