Let us try and understand how Word2Vector actually works before looking at distances:
There are 2 ways of generating vectors for a word :
- Continuous bag of words
- Skip grams
The following diagram explains the difference between the two approaches.
In case you want to further understand the nitty gritty of these two approaches, there are tons of blogs out there.
Now let us look at what the image is trying to tell us.
For CBOW, it looks at the neighboring words to provide a probability of another word co-occurring in that space.
For Skipgram, given a word, it provides probabilities of all the other words occurring in the neighborhood.
Why is this important to know?
Without understanding what these algorithms are doing, its not quite possible to solve an NLP problem as word vectors are the primary inputs to any deep learning / machine learning algorithms.
What are the other capabilities Word Vectors provide us with?
As you have vectorized Words in a n-dimensional space, it gives you the ability to perform fast operations such as computing distances in this n-dimensional space. One such way is cosine distance, that you have outlined to have used.
Understanding how distances are computed can also help. Here is a small explanation of how Cosine similarity works :
When you look at vectors project in n-dimensional space, you find the difference in the angle between these vectors. If the angle is small, lets say 0, then cos(0) = 1, which implies the distance between these vectors is very small, thereby making them similar vectors.
Given this information, what can you infer from your results of distances between words:
- Your corpus has diverse distribution of words that could possible co-occur with a large number of words, thereby making distances between similar words not > 0.7
- Given the variation of words and possible presence of large amount of stop words in your vocabulary, its hard for a shallow neural network to provide accurate vectors for your words.
Suggestions that can help improve the vector quality:
- Using TFIDF, you can remove stop words in case they are not interesting to you.
- Pre-process your text to bring them to standard form. For example, lower casing of text, removal of special characters, stripping of spaces.
- Experiment with varying embedding sizes. Start with as little as 100, as you are interested in only finite number of words.
- Visualize your vectors using tsne / PCA and projecting them to a 2 or 3 dimensional space.
- Perform the same for different epochs to see how vectors are converging / diverging.
- Its also very important to understand your corpus well, so that you know what these vectors imply and how they co-occur.
Hope this helps.