# In a recommender system, how can you normalise the similarity between two arbitrary users?

Consider the following problem:

There are 1000 users, 100 items (movies, for example), and 10000 ratings. The probability of a user, $u$, rating a movie, $i$, is $\mathbb{P}(R_{u_i}=\text{yes})=\frac{1}{10}$, and the probability of any two users rating the same movie is $\mathbb{P}(R_{u_i}=R_{v_i}=\text{yes})=\frac{1}{100}$.

If we let a random variable $x_i=1$ if $i$ is rated by both $u$ and $v$, and $0$ otherwise, then we have $$E\Big[\sum_{i=1}^{100}x_i\Big]=100\times\big(1\frac{1}{100} + 0\frac{1}{100}+0\frac{1}{100}+ ... \big)=1.$$ The similarity between $u$ and $v$ is $$PC(u, v)=\frac{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})(R_{v_i}-\bar{R_v})}{\sqrt{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})\sum_{i\in T_{uv}}(R_{v_i}-\bar{R_v})}}=1,$$ where $T_{uv}$ is the set of all items rated by $u$ and $v$, and I used the fact that $R_{u_i}$ and $R_{v_i}$ become irrelevant in comparison to $\bar{R_u}$ and $\bar{R_v}$. This is an obvious problem because it means that in situations like this you will always have a perfect similarity between $u$ and $v$.

I believe that the normalisation solution is something like this:

$$PC(u, v)=\frac{\text{min}(|T_{uv}|, \beta)}{\beta}PC(u, v),$$ where $\beta\in[25, 50]$.

I don't understand the logic behind this normalisation. Could you explain please?

• Why don't you find a vector embedding for each user then calculate the cosine similarity, which is bounded between -1 and +1? Where did you read this normalization solution? – Emre Nov 28 '17 at 20:02
• @Emre a cosine similarity won't suffice in a real life situation because it doesn't account for users with the same preferences who rank with different 'harshness'. So it is normalised to the PC – ODP Nov 28 '17 at 20:54
• So preprocess the ratings; standardize? – Emre Nov 28 '17 at 21:00

This normalization means if the number of mutual voting of $u$ and $v$ is low and not enough proportional to total number of voting (here from $\frac{25}{10000} = \frac{1}{400}$ to $\frac{50}{10000} = 0.005$) you should apply this fact into the similarity in some way. Because, your observation is not enough and you can't talk about it confidently.