6
$\begingroup$

I'm looking for a place to find benchmarks against which to evaluate performance on public datasets.

In this instance, I'm interested in results on the MovieLens10M dataset. It seems to be referenced fairly frequently in literature, often using RMSE, but I have had trouble determining what might be considered state-of-the-art.

I did find this site, but it is only for the 100K dataset and is far from inclusive:

http://www.recsyswiki.com/wiki/MovieLens_100k_benchmark_results

$\endgroup$
  • $\begingroup$ The lowest RMSE I've found so far is contained here: users.cecs.anu.edu.au/~akmenon/papers/autorec/paper.pdf $\endgroup$ – jamesmf Aug 27 '15 at 3:21
  • 1
    $\begingroup$ The link in your comment is dead. $\endgroup$ – daknowles Aug 10 '17 at 5:30
  • 1
    $\begingroup$ all links are dead.. is there any benchmark/competition(in Kaggle) for MovieLens20M/latest dataset? $\endgroup$ – Janmejaya Nanda Mar 30 '18 at 18:21
5
$\begingroup$

As You said, the most common situation for recommender system is to predict rating. Then RMSE/MAE is used. For results of a ranked item list different measures are used, e.g. Prec@K, Rec@K, AUC, NDCG, MRR, ERR. However, the are many algorithms for recommendation with its own hyper-parameters and specific use cases. Depending what You are trying to achieve You should benchmark Your solution with suitable method. The best source for this can be publication, where as You noticed, many times results for ML dataset are given. What's more about benchmarking recommender systems, sometimes the best prediction results are not so important as other aspect e.g. novelty or diversity.

For most common methods try this benchmark results for librec project: http://www.librec.net/example.html There are no results for ML-10M, only ML-1M for rating and ML-100K for ranking. But You can always run chosen algorithm by yourself. There are other projects of this kind: http://www.mymedialite.net/, http://lenskit.org/.

Saying about evaluation of recommender systems this project should be mention: https://github.com/recommenders/rival It's dedicated for data preparation (splitting) and evaluation for recommender systems. It was presented at last RecSys'2014 [1]. And also look at this [2]. There are some benchmark results too.

[1] A. Said and A. Bellogín, “RiVal – A Toolkit to Foster Reproducibility in Recommender System Evaluation,” pp. 371–372, 2014.

[2] A. Said and A. Bellogín, “Comparative Recommender System Evaluation : Benchmarking Recommendation Frameworks,” pp. 129–136.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.