Ensure trained word embeddings get high similarity with particular words

Question

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times

entertainment films Movies cinema
entertainment Movies 
entertainment films
entertainment cinema

The idea behind using a training file like the one above is to ensure that words like movies, etc come out to be most similar to entertainment.

>>> wv_model = gensim.models.Word2Vec(sents, size=300, min_count=1, 
    workers=8, window=1, sg=0)

But when I check the results I entertainment actually has a negative similarity score

>>> wv_model.most_similar(positive=['Movies'])
[('cinema', 0.14602532982826233), ('films', -0.022810805588960648), ('entertainment', -0.030070479959249496)]

The result I am trying to achieve is to ensure that the most similar word for movies, films, cinema comes out to be entertainment

Answer 1

Word2vec (or, e.g. GloVe) learn similar embeddings for words that occur in *similar contexts (i.e. co-occur with the same distribution of words) not words co-occurring directly.$^*$ This makes sense intuitively since words that appear together aren't necessarily similar (e.g. synonymous), but may be otherwise related, e.g. blue and sky.

e.g. say film and movie each co-occur a lot with words like actor, director, rating, etc then their embeddings will be similar (relative to those of other words) - even if movie and film are never seen together (which is often the case for synonyms as you would say one or the other).

[$^*$ of course, if two words frequently appear together then their contexts may well be similar.]

To deliberately force embeddings of two words $w_1, w_2$ to be similar is not trivial (unless you manually change their embeddings..) since the embeddings capture statistics from the corpus. For word2vec to learn similar embeddings, the distributions of words that co-occur with $w_1, w_2$ must be similar. One way(/hack) to do that could be to duplicate the corpus and in the second copy replace all instances of $w_1$ with $w_2$ and vice versa. Then train on both copies joined together.

[More detail if interested: Words that do directly co-occur typically have a higher dot product between their embeddings since that approximates their associated pointwise mutual information (e.g. see this paper for info). However, similarity is estimated by the cosine of the angle between embeddings (1 if identical, 0 if orthogonal, etc), which checks if embeddings are in the same direction. Since the embedding of a word reflects the distribution of all words that co-occur with it (e.g. see this paper), embeddings having similar directions tends to indicate words that have similar co-occurrence distributions, i.e. similarity.]