9
$\begingroup$

While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean distance suitable for this task.

Let us consider 2 vectors a and b. Where, a = [-1,2,-3] and b = [-3,6,-9], here b = 3*a, i.e, both the vectors have same direction but different magnitude. The cosine similarity between a and b is 1, indicating they are identical. While the euclidean distance between a and b is 7.48.

Does this mean the magnitude of the vectors is irrelevant for computing the similarity in the word vectors?

$\endgroup$

1 Answer 1

16
$\begingroup$

You're asking two questions here.

  1. Does this mean the magnitude of the vectors is irrelevant?

Yes. Cosine similarity is $ S_{cos} = \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the definition of inner product, $A \cdot B = \|A\|\|B\|\cos\theta$. Cosine distance is simply $1 - S_{cos}$.

  1. Why is the cosine distance used? Or, to think of it another way, why is the answer to (1) a desirable property to have in a distance metric?

In a word embedding, we choose a dimensionality $d$ for the embedding. This is the number of components in our embedding space. The components (or, linear combinations of the components) are meant to encode some kind of semantic meaning. Classic examples are like that the vector for "queen" plus the vector for "man" should be near the vector for "king". That sort of thing. There's a direction that roughly corresponds to "royalty" and a direction for gender.

Look at your example, where $b = 3a$, $a=[-1,2,-3], b=[-3,6,-9]$. This is a perfect illustration of why we use cosine similarity. They have very different magnitudes but point in the same direction. They have cosine distance 0, and we want that because it means that they have the same relative proportion of each component.

If we use euclidean distance, $a$ and $b$ are $\sim7.48$ units apart. It would be easy to find another vector $c$ that is around the same distance from $a$ as $b$ is, in a completely different direction. If our space has been learned properly, $c$ should have completely different semantic meaning from $b$, but they're both the same distance from $a$. The euclidean distance doesn't measure the similarity that we want very well.

$\endgroup$
3
  • 1
    $\begingroup$ This is a great answer! Well done man. I think you have a typo though. They vectors a and b have cosine distance 0 and similarity 1. $\endgroup$ Commented Aug 31, 2022 at 9:19
  • 1
    $\begingroup$ Yes! Edited to fix, thanks for pointing that out! $\endgroup$
    – Matthew
    Commented Sep 1, 2022 at 12:33
  • 1
    $\begingroup$ Thanks Matthew. This was a very good explanation. If you happen to see this 6 months later, I still haven't grasped why only factoring in the cosine similarity is important. Using example above, b = 3a. But what if we have a vector c = 1000a. In this case the cosine distance is zero but the scalar magnitude is significantly different. I would have thought this would be an issue for word embeddings. Any additional explanation would be massively appreciated. $\endgroup$ Commented Feb 26, 2023 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.