Suppose you are given two Recommender Systems to evaluate, A
and B
. Model A
is trained with large data, and model B
with small data (this implies that A
would have a larger pool of items to pick for recommendations).
How would you compare the two models? One strategy would be to calculate precision and recall in the scenario where both models are subjected to their data sets ('A' with big data, 'B' with small data) with a 80/20 split, and then calculate precision and recall. However, I'm not sure if the precision and recall results are comparable in this case. What do you think?
Another approach would be to train A
with big data, train B
with small data, but fix the test set (meaning, the test set would be the same for both A
and B
). But isn't this "unfair", given that model A
is based on big-data, and therefore has a larger pool of items to recommend from?
How would you compare the two models?