I'm trying to map similar ngrams using Wordnet and synsets. For example: elder brother and older sibling should map to the same entity.

What would be the best way to implement this? I've been thinking and so far I've only come up with a brute-force approach checking for each synset of each word and trying to find a similar word or else add them as a new entity.

I am wondering whether there are any better methods of implementing this?

for 1-gram:

from nltk.corpus import wordnet as wn
from nltk.stem import WordNetLemmatizer
l = WordNetLemmatizer()

older = 'older'
elder = 'elder'
older_lemma = l.lemmatize(older, pos=wn.ADJ)
elder_lemma = l.lemmatize(elder, pos=wn.ADJ)

for syn in wn.synsets(older_lemma):
    if elder_lemma in syn.lemma_names():

Ideally, I would want to extend it to n-grams and I'm searching for a better way to do this.

Edit : I'm not looking for vector based solutions.

What I'm thinking of is some sort of crude but fast similarity algorithm that can give me a crude representation of how close can two words/synsets possibly be. That way I could eliminate most of the absolutely dissimilar words saving time. I'm not sure if it exists


You may want to check word2vec algorithm which can map words to a semantic vector space such that words used in similar context will be closer to each other. You can then use cosine similarity on word vectors for calculating similarity.

For extending the similarity comparison to multiple words or sentences, you can look at word mover distance after training word2vec model.

The downside of the above approach is that you need a large text corpus for training word2vec model. You can either use something specific to your domain or use open source datasets (news articles, Project Gutenberg books, wikipedia dumps).

| improve this answer | |
  • $\begingroup$ Thanks but I'm aware of word2vec. I came across while researching for this project. However, I want to try an approach that doesn't rely on vectors instead is based upon the bagofwords/text based model which is why I'm trying to avoid word2vec or GloVe or FastText $\endgroup$ – Naman Aug 17 '18 at 8:51
  • $\begingroup$ Can you add this to the question that you are not looking for w2v or semantic vector based solutions? $\endgroup$ – hssay Aug 17 '18 at 9:07
  • $\begingroup$ Have you tried wup_similarity and path_similarity available with nltk/wordnet? $\endgroup$ – hssay Aug 17 '18 at 9:21
  • $\begingroup$ I have but in my very limited experience, I found it to be very very slow. Let's say I'm running this on a 200 word essay, that would be very slow. I don't know what the time complexity for wup similarity is. In the end if nothing better works out, wup is one option. Looking to see if there are any other better methods $\endgroup$ – Naman Aug 17 '18 at 9:37

The best solution is definitely using word vectors.

  • Create your own with Keras' Embedding() layers, for example: maybe powerful, but the pehaps the slowest solution.
  • Create your own with gensim library: very quick and simple, I'd go for that.
  • Dowload Google's pretrained Glove embeddings and apply them directly: quick application, but the whole file of pretrained vectors is very large.

Once you have trained embeddings, you can represent each ngram as a vector of word embeddings, with shape:

( number of words in ngram , embedding size )

If you are looking at some very crude but fast solution, you could then average the ngram vectors into one, and compute Euclidean distance metrics between them. That's the fastest way to deal with the problem IMHO.

  • . - . - . - . -


Another fast solution you can do directly in gensim is to train a doc2vec from scratch. In that way, you'd immediately get an embedding vector for the whole document (i.e. ngram). I've never tried doc2vec on small pieces of text such as ngrams though, baymbe it's worth to give it a try. In gensim it's few lines of code.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.