An old question: Cosine or Euclidean to compute similarity of embeddings?

Question

Lately I heard a question in a NLP interview. The question is about why use Cosine similarity to compute similarity between embeddings (Dense Embeddings - which I think produced by Deep Neural Netwrok).

Here is what I thought:

• The metric is used to compute similarity between embeddings (in Inference mode) based on what the model had been trained. E.g the model might be trained using some metric learning loss functions like: triplet loss and contrastive loss, and the metric is cosine, then of course we use cosine in Inference mode.

• What about the case that the model is trained base on softmax loss (without using distance metric in computing the score) for some classification task. Then does cosine or euclidean makes any sense in this case?

Answer 1

There are a few reasons to use cosine similarity

1. Metric normalization

Cosine similarity is always bounded between [-1, 1], regardless of the magnitude of the vectors involved. This makes it easy to compare across similarity measurements.

Euclidean distance on the other hand is unbounded. This can make it difficult to compare different distance measurements

2. Magnitude invariance

Cosine similarity is invariant to vector magnitude. Similar to metric normalization, this makes it easier to compare across similarity measurements.

This is also useful if your embedding process has systemic factors influencing embedding magnitude. For example, factors like how often a certain token appears or how long a document is can influence embedding magnitude (depending on the embedding method used). You can end up with embeddings that represent the same semantic concept with very different magnitudes.

More broadly, you encounter a lot of cases where embeddings of semantically similar documents/images/whatever have similar direction but different magnitude. Cosine similarity can handle these situations, while euclidean distance can't.

3. Ease of compute

If you normalize your embeddings, cosine similarity can be computed with a simple dot product. This is marginally faster than euclidean distance, but it adds up when you're using embeddings at scale.

4. Training decisions

As you mention, if you make the explicit decision to train an embedding model with cosine similarity, you then have to use the same metric for

5. A wrong reason

When this topic comes up, you typically hear something along the lines of "euclidean distance suffers from the curse of dimensionality, so you use cosine similarity instead". The first part is correct, but cosine similarity does not solve the problem.

As an experiment: choose a dimensionality d. Randomly sample d-dimensional pairs of points from a normal distribution. Measure the euclidean distance and cosine similarity between the points. Repeat for a large number of pairs. Compute the variance of the two metrics.

What you will find is that as d increases, the variance of both metrics goes to 0. Euclidean distance approaches a constant value while cosine similarity approaches 0 (ie everything is orthogonal to everything else). Both metrics suffer from the curse of dimensionality issue.

That said, cosine similarity empirically works better with higher dimension embeddings