0
$\begingroup$

Consider the following problem:

There are 1000 users, 100 items (movies, for example), and 10000 ratings. The probability of a user, $u$, rating a movie, $i$, is $\mathbb{P}(R_{u_i}=\text{yes})=\frac{1}{10}$, and the probability of any two users rating the same movie is $\mathbb{P}(R_{u_i}=R_{v_i}=\text{yes})=\frac{1}{100}$.

If we let a random variable $x_i=1$ if $i$ is rated by both $u$ and $v$, and $0$ otherwise, then we have $$E\Big[\sum_{i=1}^{100}x_i\Big]=100\times\big(1\frac{1}{100} + 0\frac{1}{100}+0\frac{1}{100}+ ... \big)=1.$$ The similarity between $u$ and $v$ is $$PC(u, v)=\frac{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})(R_{v_i}-\bar{R_v})}{\sqrt{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})\sum_{i\in T_{uv}}(R_{v_i}-\bar{R_v})}}=1,$$ where $T_{uv}$ is the set of all items rated by $u$ and $v$, and I used the fact that $R_{u_i}$ and $R_{v_i}$ become irrelevant in comparison to $\bar{R_u}$ and $\bar{R_v}$. This is an obvious problem because it means that in situations like this you will always have a perfect similarity between $u$ and $v$.

I believe that the normalisation solution is something like this:

$$PC(u, v)=\frac{\text{min}(|T_{uv}|, \beta)}{\beta}PC(u, v),$$ where $\beta\in[25, 50]$.

I don't understand the logic behind this normalisation. Could you explain please?

$\endgroup$
  • $\begingroup$ Why don't you find a vector embedding for each user then calculate the cosine similarity, which is bounded between -1 and +1? Where did you read this normalization solution? $\endgroup$ – Emre Nov 28 '17 at 20:02
  • $\begingroup$ @Emre a cosine similarity won't suffice in a real life situation because it doesn't account for users with the same preferences who rank with different 'harshness'. So it is normalised to the PC $\endgroup$ – ODP Nov 28 '17 at 20:54
  • $\begingroup$ So preprocess the ratings; standardize? $\endgroup$ – Emre Nov 28 '17 at 21:00
1
$\begingroup$

This normalization means if the number of mutual voting of $u$ and $v$ is low and not enough proportional to total number of voting (here from $\frac{25}{10000} = \frac{1}{400}$ to $\frac{50}{10000} = 0.005$) you should apply this fact into the similarity in some way. Because, your observation is not enough and you can't talk about it confidently.

In other words, it can be confident factor which you apply into the similarity measure.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.