how to evaluate top n recommendation system with movie lens dataset?

Question

Based on my research a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And I'm currently developing a collaborative filtering recommendation system, and basically is recommends the top 'n' items to a user (I've used the user-item algorithm).

So on top of that, I will try to evaluate my recommendation system using the movie lens dataset, and basically from my research typical evaluation measures for top-N recommendation are normalized discounted cumulative gain (NDCG) and precision/recall. So my question is how can I evaluate using these metrics (or if you have any other suggestion for another metric) using the movie lens dataset (since it's a rating of users to items).

Thanks. Any suggestions are welcome.

Answer 1

For various metrics feel free to look at various benchmarking libraries including MyMediaLite and LibRec. If you are doing a TOP N approach, then the way to evaluate this using a Movielens system is simple convert the ratings into binary likes and dislikes based on some threshold. Essentially you would take the "likes" of a user. Find the user is the testing set. Find the number of items that the user likes and see if the recommendation guesses the missing data correctly.

The key to thinking about these datasets is that the fact that they chose to rate a movie at all means the user actually went out an saw the movie. The fact that they saw the movie indicates an implicit approval of the basic premise. As such, a good recommender should be able to predict which items that user rates in the future.

The easiest way to evaluate the recommendation system though is just to plug into an evaluation framework such as LibRec, MyMediaLite, or various other ones and compare the returned metrics to your algorithm. These libraries typically have code that will generate the testing, training, and evaluation subsets for you and all you have to do is hook up your code as a module.

As a bonus, feel free to add your recommendation algorithm as a pull request to the library so other researchers can try it too and you'll have an excellent open source implementation of your algorithm that future researchers can reference in their own work.