How do I perform Leave One Out Cross Validation For Top n Recommendation Sytems?

Question

I am new in making recommendation systems . I am using the surpriselib library to evaluate my recommendations. All the Accuracy Metrics are well supported in this library. But I also want to compute the Hit Rate of my top n recommender system.

I know the formula for hit rate is:

(no items users have already purchased)/(no of users)

But this does not makes sense to me because to train and test the user vs item ratings I have only those users in my dataset which have already rated all the items that are in the dataset. (**After all this is what enables me to measure accuracy of my rating predictions **)

This means I will get a HIT RATE OF 100% everytime !!!

Is my theory wrong if not what can I do to properly measure hit rate I know there is one method called Leave one out cross validation which can help me in this case . If that's the answer how do they work with top n recommender systems?

Answer 1

Here is my procedure for computation of the "Hit Rate with leave-one-out cross validation":

1. leave only one actual user-item interaction (this can be rating, purchase or whatever you are dealing with in your data) out of the train dataset (this gives you a total of U "unseen" interactions provided that you have at least one interaction per user);
2. Generate top n recommendations for all user with whatever recommendation system you are using;
3. if a user actually interacted with the recommended item (is the left-out" item was among the top n recommendations for that particular user) => that's a hit Hit rate = sum of all the hits across the users (this can be at most U since you can have at most 1 hit per user) DIVIDED BY the number of users (U).