I need to process natural language sentences in which words can appear with morphological variations: car -> cars; play -> playing, played; etc. There might be hyphens also, e.g. "dog-friendly hotel", "load-bearing walls", "rock-hard cake", etc. Are there any SOTA techniques in NLP for handling morphological variations in words in a robust way?

I was thinking that maybe working at the character level could help to learn morphological patterns in words (e.g. "ing" suffix at the end of a verb means that it's a gerund, etc.), but that would require using some sort of character-level embeddings instead of the typical word-level embeddings which are commonly used. So I'm not sure if this is the best approach, and I'd rather be informed of the current state-of-the-art before proceeding.

Motivating context: I'm working on VQA (visual question answering), and I'm still trying to figure out how to best implement the NLP branch that will process the questions. I want to implement a robust model, that's why ideally I would like it to be robust to morphological variations in words, which happen all the time in natural language.

Note: although my motivating context is VQA, answers to this post need not be restricted to the VQA domain, so feel free to forget about VQA when answering if you want.

  • $\begingroup$ I don't completely understand the question but the traditional method is to extract the lemma with a lemmatizer (also called stemming). Many Part-of-speech (POS) taggers can output both the POS tag and the lemma. $\endgroup$ – Erwan Jul 19 '19 at 15:15
  • $\begingroup$ @Erwan by using stemming wouldn't you be losing the morphological information in words? for instance, verb conjugation (past, present, future, third-person, first-person), singular vs plural, etc.? $\endgroup$ – Pablo Messina Jul 21 '19 at 4:22
  • $\begingroup$ You can keep both for every token: POS tag + lemma. $\endgroup$ – Erwan Jul 21 '19 at 11:16

You can use WordNet for morphy

from nltk.corpus import wordnet as wn

>>> print(wn.morphy('dogs'))
|improve this answer|||||
  • $\begingroup$ but if you do that then you are losing the 's', so your model cannot distinguish between singular and plural $\endgroup$ – Pablo Messina Jun 20 '19 at 20:22
  • $\begingroup$ My bad, I misunderstood the question. Yes, you are right. we don't apply these techniques for word embeddings. $\endgroup$ – nag Jun 21 '19 at 4:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.