Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm looking for a place to find benchmarks against which to evaluate performance on public datasets.
In this instance, I'm interested in results on the MovieLens10M dataset. It seems to be referenced fairly frequently in literature, often using RMSE, but I have had trouble determining what might be considered state-of-the-art.
I did find this site, but it is only for the 100K dataset and is far from inclusive:
http://www.recsyswiki.com/wiki/MovieLens_100k_benchmark_results
As You said, the most common situation for recommender system is to predict rating. Then RMSE/MAE is used. For results of a ranked item list different measures are used, e.g. Prec@K, Rec@K, AUC, NDCG, MRR, ERR. However, the are many algorithms for recommendation with its own hyper-parameters and specific use cases. Depending what You are trying to achieve You should benchmark Your solution with suitable method. The best source for this can be publication, where as You noticed, many times results for ML dataset are given. What's more about benchmarking recommender systems, sometimes the best prediction results are not so important as other aspect e.g. novelty or diversity.
For most common methods try this benchmark results for librec project: http://www.librec.net/example.html There are no results for ML-10M, only ML-1M for rating and ML-100K for ranking. But You can always run chosen algorithm by yourself. There are other projects of this kind: http://www.mymedialite.net/, http://lenskit.org/.
Saying about evaluation of recommender systems this project should be mention: https://github.com/recommenders/rival It's dedicated for data preparation (splitting) and evaluation for recommender systems. It was presented at last RecSys'2014 [1]. And also look at this [2]. There are some benchmark results too.
[1] A. Said and A. Bellogín, “RiVal – A Toolkit to Foster Reproducibility in Recommender System Evaluation,” pp. 371–372, 2014.
[2] A. Said and A. Bellogín, “Comparative Recommender System Evaluation : Benchmarking Recommendation Frameworks,” pp. 129–136.
