word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7

Question

I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), although I'm not using its character level n-grams, so it's basically just word2vec.

I'm testing the results by looking at some of the "most similar" words to key and the model seems to be working very well, except that the most similar words get at most a similarity score (using cosine similarity, gensim's FastText most_similar() function) of 0.7; versus google's or spacy's embeddings synonyms tend to have ~0.95+ similarity. I'm wondering why they don't in my case. This is not necessarily a problem, but it is not clear to me why this would happen; how come even very similar words, with very similar uses only get 0.7 in similarity? Again, the similar words themselves do make a lot of sense, so the training went well, but the score is kind of low.

My best guess is that maybe the embedding size I'm using is too big - 256. Can anyone suggest other explanations?

I trained it using CBOW, 15 epochs, window size=4, a few tens of millions of documents but only ~100000 words in the vocabulary that I'm interested in.

Answer 1

Let us try and understand how Word2Vector actually works before looking at distances:

There are 2 ways of generating vectors for a word :

1. Continuous bag of words
2. Skip grams

The following diagram explains the difference between the two approaches.

In case you want to further understand the nitty gritty of these two approaches, there are tons of blogs out there.

Now let us look at what the image is trying to tell us.

For CBOW, it looks at the neighboring words to provide a probability of another word co-occurring in that space.

For Skipgram, given a word, it provides probabilities of all the other words occurring in the neighborhood.

Why is this important to know?

Without understanding what these algorithms are doing, its not quite possible to solve an NLP problem as word vectors are the primary inputs to any deep learning / machine learning algorithms.

What are the other capabilities Word Vectors provide us with?

As you have vectorized Words in a n-dimensional space, it gives you the ability to perform fast operations such as computing distances in this n-dimensional space. One such way is cosine distance, that you have outlined to have used.

Understanding how distances are computed can also help. Here is a small explanation of how Cosine similarity works :

When you look at vectors project in n-dimensional space, you find the difference in the angle between these vectors. If the angle is small, lets say 0, then cos(0) = 1, which implies the distance between these vectors is very small, thereby making them similar vectors.

Given this information, what can you infer from your results of distances between words:

1. Your corpus has diverse distribution of words that could possible co-occur with a large number of words, thereby making distances between similar words not > 0.7
2. Given the variation of words and possible presence of large amount of stop words in your vocabulary, its hard for a shallow neural network to provide accurate vectors for your words.

Suggestions that can help improve the vector quality:

1. Using TFIDF, you can remove stop words in case they are not interesting to you.
2. Pre-process your text to bring them to standard form. For example, lower casing of text, removal of special characters, stripping of spaces.
3. Experiment with varying embedding sizes. Start with as little as 100, as you are interested in only finite number of words.
4. Visualize your vectors using tsne / PCA and projecting them to a 2 or 3 dimensional space.
5. Perform the same for different epochs to see how vectors are converging / diverging.
6. Its also very important to understand your corpus well, so that you know what these vectors imply and how they co-occur.

Hope this helps.