34
$\begingroup$

It looks like the cosine similarity of two features is just their dot product scaled by the product of their magnitudes. When does cosine similarity make a better distance metric than the dot product? I.e. do the dot product and cosine similarity have different strengths or weaknesses in different situations?

$\endgroup$
  • $\begingroup$ Note that neither of these are proper distance metrics, even if you transform them to be a value that is small when points are "similar". It may or may not matter for your use case. $\endgroup$ – Sean Owen Jul 18 '14 at 11:34
36
$\begingroup$

Think geometrically. Cosine similarity only cares about angle difference, while dot product cares about angle and magnitude. If you normalize your data to have the same magnitude, the two are indistinguishable. Sometimes it is desirable to ignore the magnitude, hence cosine similarity is nice, but if magnitude plays a role, dot product would be better as a similarity measure. Note that neither of them is a "distance metric".

$\endgroup$
  • 3
    $\begingroup$ "distance metric" is commonly used as an opposite of "similarity" in literature: the larger distance, the smaller similarity, but basically they represent same idea. $\endgroup$ – ffriend Jul 17 '14 at 20:52
  • 1
    $\begingroup$ @ffriend You mean 'dissimilarity'. Metric has a precise definition. $\endgroup$ – Memming Mar 29 '16 at 12:42
7
$\begingroup$

You are right, cosine similarity has a lot of common with dot product of vectors. Indeed, it is a dot product, scaled by magnitude. And because of scaling it is normalized between 0 and 1. CS is preferable because it takes into account variability of data and features' relative frequencies. On the other hand, plain dot product is a little bit "cheaper" (in terms of complexity and implementation).

$\endgroup$
  • $\begingroup$ Why the dot product alone (equivalent to not normalizing) not account for features' data and frequency? I don't know that this is the difference. $\endgroup$ – Sean Owen Jul 18 '14 at 11:27
  • 1
    $\begingroup$ Perhaps, I wasn't clear. I was talking about data diversity. E.g., we have two pairs of documents. Within each pair docs are identical, but pair-1 documents are shorter, than pair-2 ones. And we computing similarity within each pair. Dot product would produce different numbers, though in both cases maximum similarity estimate is expected. $\endgroup$ – sobach Jul 18 '14 at 12:36
5
$\begingroup$

I would like to add one more dimension to the answers given above. Usually we use cosine similarity with large text, because using distance matrix on paragraphs of data is not recommended. And also if you intend your cluster to be broad you tend to go with cosine similarity as it captures similarity overall.

For example if you have texts which are two or three words long at max I feel using cosine similarity does not achieve the precision as achieved by distance metric.

$\endgroup$
4
$\begingroup$

There is an excellent comparison of the common inner-product-based similarity metrics here.

In particular, Cosine Similarity is normalized to lie within [0,1], unlike the dot product which can be any real number, but, as everyone else is saying, that will require ignoring the magnitude of the vectors. Personally, I think that's a good thing. I think of magnitude as an internal (within-vector) structure, and angle between vectors as external (between vector) structure. They are different things and (in my opinion) are often best analyzed separately. I can't imagine a situation where I would rather compute inner products than compute cosine similarities and just compare the magnitudes afterward.

$\endgroup$
  • $\begingroup$ "Cosine Similarity is normalized to lie within [0,1]" It still has a dot product in the numerator, I think the range should instead be [-1, 1]? $\endgroup$ – Kari Mar 19 '18 at 9:10
2
$\begingroup$

From a geometric point of view, if all your data are unitary, $\forall x, ||x||^2 = \langle x,x \rangle = 1$, then the scalar product of two vectors defines an angle $\phi$, $\langle x,y \rangle = \cos \phi$, and you have a distance $\phi = \arccos \langle x,y \rangle$.

Visually, all your data live on a unit sphere. Using a dot product as a distance will give you a chordal distance, but if you use this cosine distance, it corresponds to the length of the path between the two points on the sphere. That means, if you want an average of the two points, you should take the point in-between on this path (geodesic) rather than the mid-point obtained from the 'arithmetic average/dot product/euclidean geometry' since this point does not live on the sphere (hence essentially not the same object)!

$\endgroup$
1
$\begingroup$

As others have pointed out, these are not distance "metrics", because they do not satisfy the metric criteria. Say instead "distance measure".

Anyway, what are you measuring and why? That information will help us give a more useful answer for your situation.

$\endgroup$
  • $\begingroup$ I've always wondered about the difference between measures and metrics. According the government (NIST): "...We use measure for more concrete or objective attributes and metric for more abstract, higher-level, or somewhat subjective attributes. ... Robustness, quality (as in "high quality"), and effectiveness are important attributes that we have some consistent feel for, but are hard to define objectively. Thus these are metrics." But the context is software engineering, not mathematics. What's your take? $\endgroup$ – ahoffer Aug 5 '14 at 20:55
  • 1
    $\begingroup$ Wikipedia was more helpful. distance(x,y) is must be non-negative; d(x,y)=0 only if x=y; d(x,y) = d(y,x); and satisfy triangle inequality- d(x, z) ≤ d(x, y) + d(y, z) $\endgroup$ – ahoffer Aug 5 '14 at 21:03
  • 1
    $\begingroup$ That's pretty much it: a metric has to meet certain axioms and a measure is less strictly defined. $\endgroup$ – sintax Aug 6 '14 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.