I'm currently reading
Hu, Koren, Volinsky: Collaborative Filtering for Implicit Feedback Datasets
One thing that confuses me is the "expected percentile ranking", an function the authors define to evaluate the goodness of their recommendations. They define it in the Evaluation methodology on page 6 as:
$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}$$
where $u$ is a user, $i$ is an item (e.g. a TV show), $r_{ui} \in [0, \infty)$ is the amount how much user $u$ did watch show $i$. $\text{rank}_{ui} \in [0, 1]$ is the percentage rank of item $i$ for user $u$. For example, it is 0 if for user $u$ the item $i$ has the highest $r$ value and 1 if the item $i$ for user $u$ has the lowest $r$ value.
I'm not super sure if I understood it correctly.
The authors write that lower values of $\overline{\text{rank}}$ are more desirable and for random predictions would lead to an expected value of $\overline{\text{rank}}$ of 0.5.
Examples
- Assume there is only one item. In this case $\text{rank} = 0$. Makes sense, as there cannot be any predictions.
- Assume there is only one user and two items with $r_{1,1} = 1$ and $r_{1,2} = 2$. Then:
$$\overline{\text{rank}} = \frac{1 \cdot \text{rank}_{1, 1} + 2 \cdot \text{rank}_{1, 2}}{1+2}$$
This means $\overline{\text{rank}} \in \{2/3, 1/3\}$.
- If there is only a single user and all $|I|$ values of $r_{ui}$ are the same, then $\overline{\text{rank}} = \sum_{ui} \text{rank}_{ui} = \frac{|I|}{2}$
Questions
- Is my understanding of the metric correct? Especially my last example and the statement by the authors that $\overline{\text{rank}} \geq 50\%$ indicated an algorithm is no better than random seem off.
- What is $t$?