8
$\begingroup$

I am working on a very basic book recommender system. I want to know what to do with the fields which aren't rated by the user when finding cosine similarity, should we ignore them and calculate only with the rated fields or should we mark them 0.

The book I am following says to exclude the fields since it will give wrong interpretation in case of Euclidean and Pearson Correlation but in case of Cosine Similarity, it makes all the non-rated fields to 0.

Can someone explain why is it needed to make the non-rated fields to 0 only for Cosine and not others or is there a different way to do it. (I know how making fields 0 in Euclidean and Pearson affect the output but not sure of cosine)

| improve this question | |
$\endgroup$
  • $\begingroup$ You can look at this question on stackoverflow for getting more insight about this. $\endgroup$ – janpreet singh Sep 12 '16 at 7:21
  • $\begingroup$ The answer partly answers the question as it shows that keeping not rated entries as 0 will definitely effect the corresponding similarity. Which could be (or not) used in the algorithm for recommender engine $\endgroup$ – janpreet singh Sep 12 '16 at 7:35
  • $\begingroup$ Have you tried mean-normalization? $\endgroup$ – Valentin Calomme Jan 31 '18 at 9:22
1
$\begingroup$

Implementation-side, there is a good reason to make 0 correspond to not rated. Since most users haven't rated most books, 0 will be the most common value and the cosine similarity function can use sparse matrices internally to speed up the computation. The sparse matrix shortcut is the main reason why people use cosine similarity in the first place.

On the other hand, it will throw off your recommendations if the algorithm interprets a 0 to be closer to a 1 than a 2. You do need to intentionally ignore those, but you can easily do so inside your cosine similarity implementation rather than filtering the data before applying it.

| improve this answer | |
$\endgroup$
  • $\begingroup$ 1. So as per your answer, following anything shouldn't affect the result? $\endgroup$ – divyum Jun 13 '16 at 17:38
0
$\begingroup$

Consider how cosine similarity is calculated.

Cosine similarity takes the dot product of two real vectors, and divides this value by the product of their magnitudes. By the Euclidean dot product identity, this is equal to the cosine of the angle between the two vectors. The upshot of this is a value between 1 and -1.

When the value is 1, those vectors are pointing in exactly the same direction. When the value is -1, the vectors are pointing in exactly the opposite direction (one is the negation of the other). When the value is 0, the vectors are perpendicular to one another; in other words, when the value is zero, these two vectors are as unalike in the feature space as it is possible to get.

The dot product is the sum of all the element-wise products of your two vectors. The bigger those numbers, the more they contribute to the cosine similarity.

Now, take any feature in your vector. The fifth, say. If you set this to zero in one of your vectors, the fifth element in the element-wise product of the two vectors will also be zero, regardless of its value in the other vector. When you sum up all these element-wise products, the fifth element will not have any impact on the summation. As a result, setting a value in your feature vector to zero means it doesn't make any contribution to the cosine similarity.

This is why setting a value to zero in a feature vector is equivalent to not including the feature in the calculation of cosine similarity, and does not does not distort cosine similarity.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.