What is the range of values of the expected percentile ranking?

Question

I'm currently reading

Hu, Koren, Volinsky: Collaborative Filtering for Implicit Feedback Datasets

One thing that confuses me is the "expected percentile ranking", an function the authors define to evaluate the goodness of their recommendations. They define it in the Evaluation methodology on page 6 as:

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}$$

where $u$ is a user, $i$ is an item (e.g. a TV show), $r_{ui} \in [0, \infty)$ is the amount how much user $u$ did watch show $i$. $\text{rank}_{ui} \in [0, 1]$ is the percentage rank of item $i$ for user $u$. For example, it is 0 if for user $u$ the item $i$ has the highest $r$ value and 1 if the item $i$ for user $u$ has the lowest $r$ value.

I'm not super sure if I understood it correctly.

The authors write that lower values of $\overline{\text{rank}}$ are more desirable and for random predictions would lead to an expected value of $\overline{\text{rank}}$ of 0.5.

Examples

• Assume there is only one item. In this case $\text{rank} = 0$. Makes sense, as there cannot be any predictions.
• Assume there is only one user and two items with $r_{1,1} = 1$ and $r_{1,2} = 2$. Then:

$$\overline{\text{rank}} = \frac{1 \cdot \text{rank}_{1, 1} + 2 \cdot \text{rank}_{1, 2}}{1+2}$$

This means $\overline{\text{rank}} \in \{2/3, 1/3\}$.

• If there is only a single user and all $|I|$ values of $r_{ui}$ are the same, then $\overline{\text{rank}} = \sum_{ui} \text{rank}_{ui} = \frac{|I|}{2}$

Questions

1. Is my understanding of the metric correct? Especially my last example and the statement by the authors that $\overline{\text{rank}} \geq 50\%$ indicated an algorithm is no better than random seem off.
2. What is $t$?

Answer 1

What is $t$?

It means observed $r_{ui}$ in the one-week test set (page 6-left).

Is my understanding of the metric correct?

First two examples are correct. Assuming user-item relation $r_{ui}^t$ is constant $a$ for all items in the test set, and predicted ranks are uniform across $[0, 1]$, then, the third one would be:

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}=\frac{\sum_{u,i} a \text{ rank}_{ui}}{\sum_{u,i} a}=\frac{1}{|I|}\sum_{u,i} \text{ rank}_{ui}=\frac{1}{|I|}\frac{|I|}{2}=\frac{1}{2}$$ This makes sense. Items are identical to the user, therefore no model can do better than random guessing, since there is no observed preference to help the model favor one item over the other. Of course, another assumption here is that training (4 weeks) and test (next week) sets are from the same distribution.