I have an understanding of the technical details of word2vec.

What I don't understand is why semantically similar words should have high cosine similarity.

From what I know, goodness of a particular embedding is seen in shallow tasks such as word analogy. I am unable to grasp the relationship between maximizing cosine similarity and good word embeddings

  • 1
    $\begingroup$ Like in clustering, you need to choose a distance function. Since we are working with vectors, cosine similarity feels a little more appropriate. But you could have chosen another. $\endgroup$ Jan 25, 2018 at 18:28

1 Answer 1


Training word embeddings does not rely on optimizing the cosine similarity of words. It usually relies on prediction problems. Take for instance the skipgram model: you are predicting the context of a word, given the word. Such models, project words in geometric spaces (for example the commonly used ~300 dimensions). In other words, a word is associated with 300-dimensional dense vectors. Due to the way that these vectors are learned, they capture the semantics of the words and hence similar words are close to the induced space.

Cosine similarity (or dot product) captures exactly this semantic closeness. Intuitively, we want similar words to be close, because they are similar and we hope that the space models some of the word properties and meaning.

  • $\begingroup$ Thank you. I still feel the need to paraphrase what you said and please confirm if i am right. "In word2vec we are assuming that there is some latent representation of the semantics in n-dimensional space and a word can be described in those as a distributed representation. A semantic is not a number for the gender of the subject etc. but it is something uninterpretable, but can be learned and, on these semantics, interpretable properties of the words are smeared. So now in the semantics space we expect the word vectors of words carrying similar semantics to be close" $\endgroup$ Jan 25, 2018 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.