What kind of data is not appropriate using CF to do recommendation?

Question

I am currently working on a recommendation system for daily news. At first, I evaluated all the recommender algorithms and their corresponding settings (e.g., similarities, factorizers, ...etc) implemented in Mahout. Since we want to recommend daily news for users, we use the reading behavior of each user collected two days ago as training set, data of the next day as the testing set. The evaluated RMSE is good, the best recommender is SVD+SGD, so we implemented the recommender on our system for several days of trial run.

However, the result, the actually recommended news, seems to be not very attractive for real users ("not attractive" here means, the users feel like "why you recommend this to me?"). So we decided another approach: use the tags and categories and their relationship to do the main job of recommendation, the result from CF is for just supporting.

This makes me wonder if CF if not appropriate for some kind of content. Because I also worked on movie and music recommendation, CF is a good tool. But for news, it seems not the case.

Can anyone explain why this happening, and also give some guideline about how to choose appropriate recommendation methods? Thanks:)

Answer 1

The key is establishing a proper validation metric.

I notice you talk about how you tried different recommendation algorithms, but at the end of the day you evaluated them all with RMSE. But there's no particular reason to believe that minimizing RMSE generates a "subjectively better" recommendation experience for the user - it just happens to be convenient, and happens to work well in some industries, but there is no real reason why it must.

RMSE is measuring how well your recommender algorithm is predicting user behavior. But that's not the same as measuring recommendation quality. Maybe users value something else - familiarity, or serendipity, or some other quality of the item being recommended. Users don't really care about being predicted.

Given your results, if you want to understand your users further, I'd focus my efforts in coming up with a mathematical metric that more closely matches the target you care about - user satisfaction - rather than RMSE. Once you know what metric you're trying to optimize, the algorithm to optimize it is much easier to select!

Answer 2

Different domains might require different approaches, the news domain being one of the more elusive ones. News items are extremely short-lived (one, maybe two days): the very items the user viewed two days ago are just not interesting anymore. Also, CF works well if a large amount of items have "enough" events (that is, lots of users have viewed/rated them), and depending on the size of your data, most news items just don't collect enough views during their lifetime for classical CF methods to be effective.

For this reason, news recommender systems tend to use some combination of content-based filtering (the tags and categories you mention), popularity-based recommendation (this often discarded approach might work well for news), and recency scoring. There is some literature around it, try googling for "news recommender system".