currently my company's planning to use a new Recommender tool/library for a book website, and now we want to compare the result between these two tools (both of the tool use Universal Recommender System, an algorithm work on multi-events like Purchase, Wishlist, View,...). My manager said we won't have enough resources for the A/B test or online survey, so we want to do an offline evaluation.

Our idea is to run these tools on the data of a month (for example October ~ train data), and then compare these predictions' results to the real data of the next month (November ~ test data). For example:

  • Prediction Result of Tool A: u1 (user1): [i1 (item1), i3, i4], u2:....
  • Prediction Result of Tool B u1: [i2, i4, i5], u2:...
  • Real data on November: u1: {Purchase: [i4,i5], Wishlist: [i3], View: [i2]}, u2:...

Now how to evaluate which tool is better in this situation? My idea is, if a tool has a "hit" (predict the right item), then give that tool some scores, which are equal to 1 x Number of Hit Items for Purchase, 0.8x... for Wishlist, and so on. In the end, I will calculate the sum of all events and all users for a tool and them compare.

In the above example, the score will be

  • Tool A: 0.8x1 (i3 hit at Wishlist event) + 1x1 (i4 hit at Purchase event) = 1.8
  • Tool B: 0.5x1 (i2 hit at View event) + 1x2 (i4, i5 both hit at Purchase event) = 2.5

Is my method ok? Or is there any good method out there for a Multi-event recommender system like this? Because when I read some research on the internet, most of them are only for 1-event only like rating, and purchase.


1 Answer 1


Because tool A and B might result in recommendations with different numbers of items, using ratio is more suitable than the actual score.

  1. calculate hit ratio for each one

    eg. hit ratio = 1 * ( Purchase hit count / total recommended count ) + 0.8 * ( Wishlist hit count / total recommended count )

  2. use an average of hit ratio to evaluate recommendation performance


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.