I'm trying to semantically cluster polysemous words or word with different meanings in a corpus for my class study and I want to do it by word embeddings but I have no Idea how to reach to the clustered target that I want. (a similar target that I'm looking for is posted below as an image)

What I have: a corpus

What I want: clustering of K frequent words with other most related semantically similar words in a corpus considering that these words have multiple senses.

For example: suppose word cell is repeated 5000 times in a corpus, here are some sentences: "There are many organelles in a biological cell" , "He went to prison cell" and
"we are running out of cell phones", in every sentence we are receiving contextually a different meaning from cell, respectively, blood cell, prison and mobile/phone.

So I want to cluster each word [for example cell in here] with their semantically similar words. (sometime similar words are synonyms)

What I've done:

  • preprocessing corpus for finding K frequent words.
  • As each meaning of word is contextualized to the correspondent sentence, I thought we could compare BERT vectors of those sentences with other sentences, but the problem is Bert compare vector to vector and different senses are dependent on their sentence but I don't know how I should correctly locate semantically similar words in sentences that are compared to the first sentence.

I searched for related papers, there was a WordNet that seems to be similar but it is not constructed with word embedding methods.

Although there are word embeddings like GloVe, FastText, Word2Vec that can bring us similar words but the context would be ignored and I didn't find anywhere that they can work semantically!

And at last there was an ELMo, ELMo word representations take the entire input sentence into equation for calculating the word embeddings. Hence, the term "cell" would have different ELMo vectors under different contexts but it is still categorizing the words by their context and not putting Words that are semantically similar in a category, this way I have no Idea how to cluster different meanings by their semantically similar words.

Plus I've checked the papers and WSD is not the thing I want, maybe Word Sense Induction clustering` seems to be more accurate but still not exact.

Here is a photo of WordNet with word Search, kindda similar to the thing I want. (you can see each cluster of word search is connected semantically to their similar group)


I'm asking here to get more ideas or maybe my intuitions are all wrong.

Thanks for your time.

Any information would be helpful and appreciated.

  • $\begingroup$ Did you already look into n-grams? $\endgroup$
    – Peter
    Nov 7, 2021 at 20:34


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.