How can conclusions be drawn from recommendation systems evaluation?

Question

From my research, a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And basically exists many type of recommendation systems, such as collaborative filtering and content-based.

One important aspect, is that evaluation is important in assessing the effectiveness of recommendation algorithms. The commonly used metrics are the mean squared error and root mean squared error, and also there exists others such as precision an recall.

My question is after I developed a recommendation system, and after evaluating using those metrics, how can I consider that my system provides good quality recommendations, or in others words, what should be the tresholds or MSE or RMSE values to be considered a good recommendation system.

Thanks. Any suggestions are welcomed.

Answer 1

"Good", I think, is based on the state of the art at the moment. So I would look at respected models from industry leaders and use their reported accuracies as a base line for what is "good": since it comes down to what is possible.

Answer 2

First of all, it's important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold.

However, you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not that small anymore. However, although the smaller the RMSE, the better, you can make theoretical claims on levels of the RMSE by knowing what is expected from your DV in your field of research. Keep in mind that you can always normalize the RMSE.

Additional Info

Decision support accuracy metrics that are popularly used are Reversal rate, Weighted errors, Receiver Operating Characteristics (ROC) and Precision Recall Curve (PRC), Precision, Recall and F-measure. These metrics help users in selecting items that are of very high quality out of the available set of items.

The metrics view prediction procedure as a binary operation which distinguishes good items from those items that are not good. ROC curves are very successful when performing comprehensive assessments of the performance of some specific algorithms.

Precision is the fraction of recommended items that is actually relevant to the user, while Recall can be defined as the fraction of relevant items that are also part of the set of recommended items. They are computed as --

Precision (P) = Correctly Recommended Items / Total Recommended Items

Recall (R) = Correctly Recommended Items / Total Useful Recommended Items

F-measure defined below helps to simplify precision and recall into a single metric. The resulting value makes comparison between algorithms and across data sets very simple and straightforward.

F-measure = 2PR / P + R

Coverage has to do with the percentage of items and users that a recommender system can provide predictions. Prediction may be practically impossible to make if no users or few users rated an item. Coverage can be reduced by defining small neighborhood sizes.

That's all. Hope it helps!

Answer 3

You've already mentioned a few metrics and I guess they work, at least from a algorithmic point of view. However, I think a lot of the initial validation would really has to be done manually.

I work on a similar problem where we are developing a model for a very new domain. We also have the additional problem of not having a lot of labelled data. We validated our results by having a lot of meetings and reviews with subject matter experts and we're also considering several options to get crowdsourced labels as a means to validate.

Answer 4

There are numerous papers on evaluating recommender systems. See, for example:

• Evaluating collaborative filtering recommender systems
• Evaluating Recommendation Systems
• Evaluating Recommender Behavior For New Users
• Evaluating recommender systems from the user’s perspective: survey of the state of the art
• Looking for “Good” Recommendations: A Comparative Evaluation of Recommender Systems