Let's say I want to compare whether one recommender system (A) is better than the other (B).
One approach is to let people rate recommendations returned by both systems.
However, there situations when I would like to evaluate my recommender system offline. One approach I considered is to collect user ratings for recommendations from system A and turn them into a test dataset. However, if the system B returns recommendations that do not appear in test dataset, that does not mean they are bad. It just means I have no rating about them.
- try to increase test dataset volume as much as possible to increase overlap between recommenders.
- limit the size of items space, let recommender select items from 500 instead of 5*10^6 items during tests.
What other approaches would you recommend suggest?