Generating synonyms or similar words from multiples word embeddings

Question

I am looking for a way to generate synonyms, using word embeddings. From one word, and from multiple words. Such as the two example below:

"word" -> Word embedding -> generate synonym of "word"

"word", "synonym of word .. "-> -> Word embeddings -> generate a synonym of both word

I am very new at this. What do you think I should use ?

I also want to use a tools that, in further work, would take into account context for word embedding generation, such as:

"sentence in including a word" -> Word embedding of word -> generate synonym of "word" in that context

I think I will start to do it with BERT... How should I start ? or which alternative should I use ?

Thanks for your help !

Answer 1

Gensim has a built in functionality to find similar words, using Word2vec. You can train a Word2Vec model using gensim:

model = Word2Vec(sentences, size=100, window=5, min_count=5, workers=4)

You can make use of the most_similar function to find the top n similar words. It allows you to input a list of positive and negative words to tackle the 'good' and 'bad' problem. You can play around with it.

model.most_similar(positive=[], negative=[], topn=10, restrict_vocab=None)

An example, provided in the documentation:

model.most_similar(positive=['woman', 'king'], negative=['man'], topn=10, restrict_vocab=None)
[('queen', 0.50882536), ...]

topn = the number of nearest neighbors you want to the input combination list of positive and negative words.

restrict_vocab = an optional integer which limits the range of vectors which are searched for most-similar values. For example, restrict_vocab=10000 would only check the first 10000 word vectors in the vocabulary order. (This may be meaningful if you’ve sorted the vocabulary by descending frequency.)

Here is the link to the documentation of what I am talking: http://man.hubwiz.com/docset/gensim.docset/Contents/Resources/Documents/radimrehurek.com/gensim/models/word2vec.html

Here is a link to how to train a word2vec model from scratch: https://radimrehurek.com/gensim/models/word2vec.html

You can also look at some other functions that come with it which allow you to find similar words just by a single vector, you can find these in the second link:

self.wv.similar_by_vector()
self.wv.similar_by_word()

Answer 2

An automatic system for finding synonyms using word embeddings is not possible. Word embeddings find co-occurrence. For example, "good" and "bad" co-occur together in a corpus, thus are near each other in the embedding space. However, "good" and "bad" are antonyms.

A copilot system could work. Word embeddings can find a set of candidate synonyms by finding the nearest words as measured by a distance metric. Then other methods (e.g., a person) can select amongst the candidates.

Answer 3

First, some background context:

• Non-contextual word embeddings (e.g. word2vec) only reflect co-occurrence statistics. The similarity between two embedded vectors may only be loosely related to their semantics (e.g. the representations for country names like "france" and "italy" may be close) or there may even be negative correlation (antonyms may be very close).

• BERT is subword-level, not word-level. This means that before going throug the network, there is a tokenization process that splits words into word pieces. Therefore, you obtain representations of pieces of words, not words themselves, e.g. for word "difficult" you may obtain a tokenization like "diff", "i", "cult". There is no direct way of obtaining a "combined representation" from the individual subword representations.

Therefore:

• I recommend you not using BERT, because you are interested at word-level information, while BERT only offert subword-level stuff.
• I recommend you to look into ELMo, which offers word-level contextual representations.