I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. PySpark provides a function called
columnSimilarities() to do that.
My question is, do I need to normalize each product's vector before using
columnSimilarities()? I read about normalisation and cosine similarity and understood that cosine similarity normalises vectors already, as if we normalize the vectors, then the cosine similarity will be just the dot product of the 2 vectors. Reference
Also, one of the answers in this question Cosine similarity versus dot product as distance metrics suggests that
Sometimes it is desirable to ignore the magnitude, hence cosine similarity is nice, but if magnitude plays a role, dot product would be better as a similarity measure. which means cosine similarity and dot product are not the same..
I'm confused about the difference and when it's good to use normalisation before calculating cosine similarity and when it's not? and what are the different ways to normalize?