In a recommender system, how can you normalise the similarity between two arbitrary users?

Question

Consider the following problem:

There are 1000 users, 100 items (movies, for example), and 10000 ratings. The probability of a user, $u$, rating a movie, $i$, is $\mathbb{P}(R_{u_i}=\text{yes})=\frac{1}{10}$, and the probability of any two users rating the same movie is $\mathbb{P}(R_{u_i}=R_{v_i}=\text{yes})=\frac{1}{100}$.

If we let a random variable $x_i=1$ if $i$ is rated by both $u$ and $v$, and $0$ otherwise, then we have $$E\Big[\sum_{i=1}^{100}x_i\Big]=100\times\big(1\frac{1}{100} + 0\frac{1}{100}+0\frac{1}{100}+ ... \big)=1.$$ The similarity between $u$ and $v$ is $$PC(u, v)=\frac{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})(R_{v_i}-\bar{R_v})}{\sqrt{\sum_{i\in T_{uv}}(R_{u_i}-\bar{R_u})\sum_{i\in T_{uv}}(R_{v_i}-\bar{R_v})}}=1,$$ where $T_{uv}$ is the set of all items rated by $u$ and $v$, and I used the fact that $R_{u_i}$ and $R_{v_i}$ become irrelevant in comparison to $\bar{R_u}$ and $\bar{R_v}$. This is an obvious problem because it means that in situations like this you will always have a perfect similarity between $u$ and $v$.

I believe that the normalisation solution is something like this:

$$PC(u, v)=\frac{\text{min}(|T_{uv}|, \beta)}{\beta}PC(u, v),$$ where $\beta\in[25, 50]$.

I don't understand the logic behind this normalisation. Could you explain please?

Answer 1

This normalization means if the number of mutual voting of $u$ and $v$ is low and not enough proportional to total number of voting (here from $\frac{25}{10000} = \frac{1}{400}$ to $\frac{50}{10000} = 0.005$) you should apply this fact into the similarity in some way. Because, your observation is not enough and you can't talk about it confidently.

In other words, it can be confident factor which you apply into the similarity measure.