currently my company's planning to use a new Recommender tool/library for a book website, and now we want to compare the result between these two tools (both of the tool use Universal Recommender System, an algorithm work on multi-events like Purchase, Wishlist, View,...). My manager said we won't have enough resources for the A/B test or online survey, so we want to do an offline evaluation.
Our idea is to run these tools on the data of a month (for example October ~ train data), and then compare these predictions' results to the real data of the next month (November ~ test data). For example:
- Prediction Result of Tool A: u1 (user1): [i1 (item1), i3, i4], u2:....
- Prediction Result of Tool B u1: [i2, i4, i5], u2:...
- Real data on November: u1: {Purchase: [i4,i5], Wishlist: [i3], View: [i2]}, u2:...
Now how to evaluate which tool is better in this situation? My idea is, if a tool has a "hit" (predict the right item), then give that tool some scores, which are equal to 1 x Number of Hit Items for Purchase, 0.8x... for Wishlist, and so on. In the end, I will calculate the sum of all events and all users for a tool and them compare.
In the above example, the score will be
- Tool A: 0.8x1 (i3 hit at Wishlist event) + 1x1 (i4 hit at Purchase event) = 1.8
- Tool B: 0.5x1 (i2 hit at View event) + 1x2 (i4, i5 both hit at Purchase event) = 2.5
Is my method ok? Or is there any good method out there for a Multi-event recommender system like this? Because when I read some research on the internet, most of them are only for 1-event only like rating, and purchase.
