I'm a newbie in machine learning and I'm currently have a project about building a collaborative filtering (user-based) product recommendation system using KNN. My data has no label, it consists of userId, productId, and rating.
I use NearestNeighbors from sklearn.neighbors, how can I evaluate? Should I split data into train and test set? Thanks in advance.


1 Answer 1


My data has no label

Well, of course it does. Your $X$ vector is (userId, productId), and the $y$ target you wish to predict is rating. It's a classic setup. In the Netflix Challenge these would be (user, film) and stars.

Divide your dataset into train + test. Some folks use a 2/3 + 1/3 split. You may want to ensure that all of a given user's examples appear exclusively in "train" or exclusively in "test".

First we should build some very simple models. Start by computing the average rating across all products, and always report that as the prediction. Score this model. No, it won't be very good, but that's our naïve baseline, what we should predict before being told anything about a given example.

Now build a model which, given productId, predicts rating. This is just per-product averages. Score your model. It will do "ok", but not great.

Do the same with per-user averages, so you know who the tough raters are.

Finally we are ready to build a more sophisticated model which takes greater advantage of the structure revealed by dataset examples. I recommend starting out with the scikit surprise library. As before, build a model, score it, and compare its performance against your existing models. There's more than one way to define which neighbors are "near".

When splitting out train and test, you may want to do it by user rather than by user + product.

Here's the motivation. Suppose that Alice always gives either 1 or 2 stars, and Bob always gives either 4 or 5 stars, and each of them randomly winds up with 7 training examples and 3 test examples. We run the risk of rewarding a learner for building just a per-user model, rather than a user + product model. Upon being challenged with a test example mentioning Alice, the learner might ignore the product as being uninformative and respond 1.5. Similarly a Bob inference might elict 4.5, even for some turkey of a product that a hundred other reviewers assigned a rating of 3 or lower. So pay attention to data leakage of that kind when splitting into folds of train, test, and even holdout.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.