Proper evaluation method for recommendation system with implicit feedback?

Question

I am trying to implement a recommendation system for a live-streaming website. Here "users" are simply the website users and "items" are streamers that they should watch. I collected data on which users follow which streamers -- however, following a streamer does not necessarily correspond to liking the streamer. Thus the setting is: I have binary implicit feedback (0 = no follow, 1 = follow) for 10k users, 2k items, and 200k interactions. I also have follow timestamps, but I have not tried to use this data yet. The models I'm using are primarily collaborative filtering, matrix factorization/latent factor models which are implemented in Implicit & LightFM (Python). My questions are: (i) how do I make a train/validation/test split, (ii) what metric is appropriate?

My understanding is that to make a train-test split we "mask" some of the observations in the interaction matrix $R$, call these observations $\{r_{ui}^t\}$ and then train our model on $R\setminus\{r_{ui}^t\}$. So we just treat the test observations as $0$ in training phase (?). Then for each user we score all of their items in the test set (or do we score all of their items in either the train or test set?), rank the items by their scores, and then calculate an evaluation metric. Please correct/validate me :) Here are my observations regarding my two main questions...

(i). In the paper [1], the authors make a train-test split based on time period. On the other hand, in [2], the authors use a leave-one-out strategy where they drop one positive user-item pair for each user. Finally, [3] generates a test set by using a random 10% sample of entries (rather than selecting random positive entries), although it isn't clear to me how this is favorable to just selecting random positive entries. I'm leaning towards the method in [2] for its simplicity and because there does not seem to be a consensus. Furthermore, none of these works mentioned using, or offered a method of making, an additional validation set to tune hyperparameters. Is it just a matter of repeating the split method twice?

(ii). For implicit feedback I understand that recall-based metrics are more important than precision. In [1,3] they use mean percentage ranking $$\overline{rank} = \frac{\sum_{u,i} r_{ui}^t rank_{ui}}{\sum_{u,i} r_{ui}^t}$$ where $rank_{ui}$ is the percentile ranking for item $i$ in user $u$'s full ranking list and $r_{ui}^t$ is the implicit feedback rating for a test observation. This seems useful when $r_{ui}$ may take on many values and one would like to respect these ratings; however, for me $r_{ui} \in \{0,1\}$. Also, this uses the entire ranking list, while perhaps a cutoff-sensitive metric such as $MAR@K$ is better for my application (?). In [2] they only report $AUC$, probably because the BPR model optimizes for $AUC$. With their leave-one-out strategy they denote $S_\text{train}, S_\text{test}$ to be the sets of positive interactions and then compute $$AUC = \frac{1}{|U|} \sum_u \frac{1}{|E(u)|} \sum_{i,j \in E(u)} \delta(\hat{r}_{ui} > \hat{r}_{uj})$$ where $|U|$ is the number of users and $E(u) = \{(i,j) | (u,i) \in S_{\text{test}} \wedge (u,j) \notin (S_\text{test} \cup S_\text{train})\}$ are the evaluation pairs for user $u$. That is, they calculate how often a positive interaction is rated higher than an unobserved interaction. This seems like it might be a favorable metric considering our interaction data is binary.

References

[1] Hu, Y., Koren, Y., & Volinsky, C. (2008, December). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263-272). IEEE.

[2] Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2012). BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618.

[3] Johnson, C. C. (2014). Logistic matrix factorization for implicit feedback data. Advances in Neural Information Processing Systems, 27, 78.

Answer 1

First off, in terms of how to make a train/validation/test split, I would only include users with 2 or more interactions, and for each user, remove the latest interaction. Use this latest interaction as a target and recommend, for each user, their top streamers. You then compare your recommendations with the target, and compute a performance metric. Depending on your performance metric of choice (next paragraph), the number of recommendations you make might differ. This gives you a training/validation split. If you want more training data, and you believe a user's interactions are not sequential in nature (eg. liking StreamerA before StreamerB displays a different preference than vice-versa), you can permute each user into more users by leaving out a different target each time. So for each streamer u, who has n_u interactions, leave out one interaction (target), and keep the rest, and do this n_u times.

Secondly, the appropriate performance metric depends on your use case. If you are going to recommend, say k, streamers for each user, then the performance metric of choice should consider only these k. In other words, for any model performance, rank k+10 is no better than rank k+20. Moreover, maybe it is only of importance that the target is part of the top-k recommendations, then some XYZ@K-metric is most appropriate. Personally, I would probably go with top-k accuracy and average this across all users. This metric then specifies, for all my users, how often, on average, is the target in their top k recommendations.