4
$\begingroup$

I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), although I'm not using its character level n-grams, so it's basically just word2vec.

I'm testing the results by looking at some of the "most similar" words to key and the model seems to be working very well, except that the most similar words get at most a similarity score (using cosine similarity, gensim's FastText most_similar() function) of 0.7; versus google's or spacy's embeddings synonyms tend to have ~0.95+ similarity. I'm wondering why they don't in my case. This is not necessarily a problem, but it is not clear to me why this would happen; how come even very similar words, with very similar uses only get 0.7 in similarity? Again, the similar words themselves do make a lot of sense, so the training went well, but the score is kind of low.

My best guess is that maybe the embedding size I'm using is too big - 256. Can anyone suggest other explanations?

I trained it using CBOW, 15 epochs, window size=4, a few tens of millions of documents but only ~100000 words in the vocabulary that I'm interested in.

$\endgroup$
3
  • $\begingroup$ For the sake of completeness, what similarity measure is used? $\endgroup$
    – Esmailian
    May 31, 2019 at 20:48
  • $\begingroup$ I'm using gensim's FastText most_similar function which uses cosine similarity. $\endgroup$
    – Oren Matar
    Jun 1, 2019 at 6:00
  • $\begingroup$ @OrenMatar: I edited that detail into the question. Please put all the question details in the question, not littered throughout the comments. This helps the question and answers get indexed for search and retrieval, both now and in the future; whereas comments don't get indexed, can't be reliably found, are ephemeral and often disappear. $\endgroup$
    – smci
    Mar 19, 2021 at 20:51

1 Answer 1

5
$\begingroup$

Let us try and understand how Word2Vector actually works before looking at distances:

There are 2 ways of generating vectors for a word :

  1. Continuous bag of words
  2. Skip grams

The following diagram explains the difference between the two approaches.

WordVectors

In case you want to further understand the nitty gritty of these two approaches, there are tons of blogs out there.

Now let us look at what the image is trying to tell us.

For CBOW, it looks at the neighboring words to provide a probability of another word co-occurring in that space.

For Skipgram, given a word, it provides probabilities of all the other words occurring in the neighborhood.

Why is this important to know?

Without understanding what these algorithms are doing, its not quite possible to solve an NLP problem as word vectors are the primary inputs to any deep learning / machine learning algorithms.

What are the other capabilities Word Vectors provide us with?

As you have vectorized Words in a n-dimensional space, it gives you the ability to perform fast operations such as computing distances in this n-dimensional space. One such way is cosine distance, that you have outlined to have used.

Understanding how distances are computed can also help. Here is a small explanation of how Cosine similarity works :

When you look at vectors project in n-dimensional space, you find the difference in the angle between these vectors. If the angle is small, lets say 0, then cos(0) = 1, which implies the distance between these vectors is very small, thereby making them similar vectors.

Given this information, what can you infer from your results of distances between words:

  1. Your corpus has diverse distribution of words that could possible co-occur with a large number of words, thereby making distances between similar words not > 0.7
  2. Given the variation of words and possible presence of large amount of stop words in your vocabulary, its hard for a shallow neural network to provide accurate vectors for your words.

Suggestions that can help improve the vector quality:

  1. Using TFIDF, you can remove stop words in case they are not interesting to you.
  2. Pre-process your text to bring them to standard form. For example, lower casing of text, removal of special characters, stripping of spaces.
  3. Experiment with varying embedding sizes. Start with as little as 100, as you are interested in only finite number of words.
  4. Visualize your vectors using tsne / PCA and projecting them to a 2 or 3 dimensional space.
  5. Perform the same for different epochs to see how vectors are converging / diverging.
  6. Its also very important to understand your corpus well, so that you know what these vectors imply and how they co-occur.

Hope this helps.

$\endgroup$
2
  • 2
    $\begingroup$ Wow what a great answer, thank you so much! I should have mentioned I did pre-process it - removed stop words+lemmatized+lower cased everything. I also used another algorithm to find "terms" that go together like "saudi arabia" or "high tech" etc - and added an underscore between them. which, for example, found "Hewlett-Packard" as a term and identified it as the closest to "hp" which i find to be a good intecation (although again it was only 0.65 similar..). I think using a smaller vector size makes most sense so I will try that now.. $\endgroup$
    – Oren Matar
    Jun 1, 2019 at 11:44
  • $\begingroup$ Replacing 2-grams like "saudi arabia" or "high tech" with single words with an underscore between them is a really bad idea that means you're no longer using word-vectors and lose all that richness based on context, you're manually creating a new set of 2-grams. It's effectively going back to tree-based classification. $\endgroup$
    – smci
    Mar 19, 2021 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.