Evaluate two Recommender models trained with different data

Question

Suppose you are given two Recommender Systems to evaluate, A and B. Model A is trained with large data, and model B with small data (this implies that A would have a larger pool of items to pick for recommendations).

How would you compare the two models? One strategy would be to calculate precision and recall in the scenario where both models are subjected to their data sets ('A' with big data, 'B' with small data) with a 80/20 split, and then calculate precision and recall. However, I'm not sure if the precision and recall results are comparable in this case. What do you think?

Another approach would be to train A with big data, train B with small data, but fix the test set (meaning, the test set would be the same for both A and B). But isn't this "unfair", given that model A is based on big-data, and therefore has a larger pool of items to recommend from?

How would you compare the two models?

Answer 1

Something seems unclear or maybe I didn't get it. Please comment if my answer is not precise enough.

A recommender engine is not limited to its training data when it comes to query. The unfairness shows up if the pool of available items to be recommended is different for those 2 engines.

Example: I train language models for calculating the similarity between a search query and the title of all websites on the web.

Training data would be the text dataset based on which I tune my language model. Small dataset results in "bad" training so in query time, it may not work well as it did not cover the whole language distribution in training phase. BUT IT STILL GIVES A SET OF ANSWERS FOR THE QUERY!

This is the point I concern you are mistaking. Recommender engines can be compared as long as the set of queries and item pool are fixed. Then the one trained on small data works poor in compare to the one which was trained on larger amount of data.

For this comparison you "need" labeled data i.e. for a set of known queries you need to know the first $n$ item to be recommended or ranks of first $n$ item or some other kind of label. But you certainly need labelled data and when you have it, the comparison will be valid.

For evaluation metrics, please have a look at this answer.