2
$\begingroup$

I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. PySpark provides a function called columnSimilarities() to do that.

My question is, do I need to normalize each product's vector before using columnSimilarities()? I read about normalisation and cosine similarity and understood that cosine similarity normalises vectors already, as if we normalize the vectors, then the cosine similarity will be just the dot product of the 2 vectors. Reference

Also, one of the answers in this question Cosine similarity versus dot product as distance metrics suggests that Sometimes it is desirable to ignore the magnitude, hence cosine similarity is nice, but if magnitude plays a role, dot product would be better as a similarity measure. which means cosine similarity and dot product are not the same..

I'm confused about the difference and when it's good to use normalisation before calculating cosine similarity and when it's not? and what are the different ways to normalize?

Any help?

$\endgroup$
2
$\begingroup$

My question is, do I need to normalize each product's vector before using columnSimilarities()?

No, you do not need to normalize each product's vector before using columnSimilarities() since it is performed within the operation already. I think your confusion comes from the fact that your considering dot product and cosine similarity to be the same. They are not. This is the cosine similarity

Dot Product is only a component of the Cosine Similarity Function Denoted as "A (dot) B"

With regards to the answer you referenced, the solution suggests using the dot product as an alternative to cosine similarity because a dot product calculation is not affected by the magnitude. The dot product doesn't use the mean as part of its calculation.

when it's good to use normalization before calculating cosine similarity and when it's not?

I would advise against using normalization before cosine similarity. However, there are other methods you can consider if cosine similarity is not returning desired results. You can use adjusted cosine similarity or dot product(as referenced in the answer you linked). Both of these measures take into account differences in magnitude. The adjusted cosine similarity subtracts the mean before calculating cosine similarity. Dot product doesn't use the mean in its calculation. Which is important in your context. For example, Users who always rate products 5 stars and users who always rate products 1 star are not equivalent.

what are the different ways to normalize?

There are a lot of ways to normalize a vector. You can use Z-score standardization, the standard min-max scaling, l1 or l2-normalization,etc etc

Further Information: Is feature normalization needed prior to computing Cosine distance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.