From my research, a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And basically exists many type of recommendation systems, such as collaborative filtering and content-based.

One important aspect, is that evaluation is important in assessing the effectiveness of recommendation algorithms. The commonly used metrics are the mean squared error and root mean squared error, and also there exists others such as precision an recall.

My question is after I developed a recommendation system, and after evaluating using those metrics, how can I consider that my system provides good quality recommendations, or in others words, what should be the tresholds or MSE or RMSE values to be considered a good recommendation system.

Thanks. Any suggestions are welcomed.


"Good", I think, is based on the state of the art at the moment. So I would look at respected models from industry leaders and use their reported accuracies as a base line for what is "good": since it comes down to what is possible.

| improve this answer | |
  • $\begingroup$ Thanks for the answer. As an alternative, I guess I can compare two or three algorithms used for the recommendation system, and evaluate them in order to see which has the better quality and performance ( based on the metrics) $\endgroup$ – John Newman Sep 25 '16 at 11:52
  • $\begingroup$ True. I think that that approach should be part of your overall process in any case. It can get you to "Good Enough", but comparing your results to industry standards gets you to an objective "Good". $\endgroup$ – grldsndrs Sep 25 '16 at 14:27

First of all, it's important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold.

However, you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not that small anymore. However, although the smaller the RMSE, the better, you can make theoretical claims on levels of the RMSE by knowing what is expected from your DV in your field of research. Keep in mind that you can always normalize the RMSE.

Additional Info

Decision support accuracy metrics that are popularly used are Reversal rate, Weighted errors, Receiver Operating Characteristics (ROC) and Precision Recall Curve (PRC), Precision, Recall and F-measure. These metrics help users in selecting items that are of very high quality out of the available set of items.

The metrics view prediction procedure as a binary operation which distinguishes good items from those items that are not good. ROC curves are very successful when performing comprehensive assessments of the performance of some specific algorithms.

Precision is the fraction of recommended items that is actually relevant to the user, while Recall can be defined as the fraction of relevant items that are also part of the set of recommended items. They are computed as --

Precision (P) = Correctly Recommended Items / Total Recommended Items

Recall (R) = Correctly Recommended Items / Total Useful Recommended Items

F-measure defined below helps to simplify precision and recall into a single metric. The resulting value makes comparison between algorithms and across data sets very simple and straightforward.

F-measure = 2PR / P + R

Coverage has to do with the percentage of items and users that a recommender system can provide predictions. Prediction may be practically impossible to make if no users or few users rated an item. Coverage can be reduced by defining small neighborhood sizes.

That's all. Hope it helps!

| improve this answer | |
  • $\begingroup$ This would be a really good answer, in terms of detail and style. Except that it concentrates on metrics that are not really suitable for recommendation systems. $\endgroup$ – Neil Slater Oct 23 '16 at 7:21
  • $\begingroup$ @NeilSlater I don't think so. They are also important. $\endgroup$ – HelloWorld Oct 23 '16 at 8:28
  • $\begingroup$ @StudentT: Can you explain how to use AUC, F1 to assess a recommendation system? Especially one that supplies results as a vector of predicted ratings? $\endgroup$ – Neil Slater Oct 23 '16 at 9:32

You've already mentioned a few metrics and I guess they work, at least from a algorithmic point of view. However, I think a lot of the initial validation would really has to be done manually.

I work on a similar problem where we are developing a model for a very new domain. We also have the additional problem of not having a lot of labelled data. We validated our results by having a lot of meetings and reviews with subject matter experts and we're also considering several options to get crowdsourced labels as a means to validate.

| improve this answer | |

There are numerous papers on evaluating recommender systems. See, for example:

  • Evaluating collaborative filtering recommender systems
  • Evaluating Recommendation Systems
  • Evaluating Recommender Behavior For New Users
  • Evaluating recommender systems from the user’s perspective: survey of the state of the art
  • Looking for “Good” Recommendations: A Comparative Evaluation of Recommender Systems
| improve this answer | |
  • $\begingroup$ Answer is not self-contained. It would be better to summarise (maybe 1 or 2 paragraphs) just one of the papers. Link to the paper is nice for continued reading, but just naming a bunch of papers is not Stack Exchange way. If you don't have time to go into more detail and still want to help OP, maybe post a comment? $\endgroup$ – Neil Slater Oct 23 '16 at 7:24
  • $\begingroup$ Lists become hard to read as comments and I don't have the time to summarize five papers, so I'll leave the answer as is; it's the only one specific to recommender systems. $\endgroup$ – Emre Oct 23 '16 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.