2
$\begingroup$

Let's say I want to compare whether one recommender system (A) is better than the other (B).

One approach is to let people rate recommendations returned by both systems.

However, there situations when I would like to evaluate my recommender system offline. One approach I considered is to collect user ratings for recommendations from system A and turn them into a test dataset. However, if the system B returns recommendations that do not appear in test dataset, that does not mean they are bad. It just means I have no rating about them.

Some alternatives:

  • try to increase test dataset volume as much as possible to increase overlap between recommenders.
  • limit the size of items space, let recommender select items from 500 instead of 5*10^6 items during tests.

What other approaches would you recommend suggest?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

There is always a bias in offline testing in recommender systems. Real evaluation happens in A/B testing. But, that should not discourage one from doing offline testing.

There is ongoing research on this topic using multi-armed bandits. I recommend reading offline testing procedures using reco-gym.

Following workshop talks about removing these biases and developing efficient offline estimators for recommender systems. They also conducted an online course on efficiently doing an offline evaluation of recommender systems.

You can also follow a video tutorial on this website on counterfactual evaluation

Improve this answer
$\endgroup$
1
$\begingroup$

Traditional offline evaluations use metrics and methodologies borrowed from machine learning and information retrieval to estimate the performance of recommendations. They could be bias ,but they are chip alternative.
Offline evaluations follow a train-test evaluation procedure :
1.split user data into the training set and the test set.
2.train recommendation algorithms on the training set.For each user:
2.1.generate a list of recommendations
2.2.test prediction accuracy or ranking effectiveness
For the part with split user data into the training set and the test set normally i remove some part of data when train recommender and then use it as a ground true.
In your case i would calculate measures for both RS and compare the result ,of course using the same data. More about evaluation could be read in https://scholarworks.boisestate.edu/cgi/viewcontent.cgi?article=2703&context=td

Improve this answer
$\endgroup$
4
  • $\begingroup$ And how do you measure Precision@10 of recommendations if only part of them appear in groundtruth data? Let say a recommender returns 10 restaurants. 6 are good, 2 are bad according to groundtruth data, and 2 you have no feedback about in groundtruth dataset. How to compare it fairly with a recommender that returns 2 bad, 1 good and 7 unrated recommendations? $\endgroup$
    – dzieciou
    Commented Oct 17, 2020 at 3:33
  • 1
    $\begingroup$ Precision =( recommended∩relevant) /recommended Recall = ( recommended∩relevant) /relevant More about metrics you can read here: medium.com/@bond.kirill.alexandrovich/… $\endgroup$
    – mariq vlahova
    Commented Oct 17, 2020 at 7:25
  • 1
    $\begingroup$ That would give a precision of 0.6 for the first system and 0.1 for the second. However, for the second system, there are still 7 recommendations that might be good but we don't know that without collecting more ratings. There are measures to estimate how much missing information might impact results: exascale.info/assets/pdf/tonon2015irj.pdf. $\endgroup$
    – dzieciou
    Commented Oct 17, 2020 at 7:43
  • $\begingroup$ very helpful article.Thanks :) $\endgroup$
    – mariq vlahova
    Commented Oct 17, 2020 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.