How to prepare the training data for SVD-based Recommendation?

Question

I am trying to build an SVD-based recommender system. According to my understanding, the training data should only contain the users who buy at least m items and the items which are bought by n unique users. Otherwise, if I use all users and items (including low-frequency users and items), I think the training data will be noisy.

However, here is the problem: To build the training data for SVD, I first retrieve all users who buy at least m items from the database. But there are some items bought by these users are not bought by at least n users. But after further filtering out these low-frequency items from the training data, some of the remaining users will not have item sets which contain m items because some items are deleted.

I feel that I am not on the right track. How am I supposed to build the training data for SVD?

Answer 1

I consider this answer as a general approach for the data preparation for CF algorithms and based on my experience. I do not know the specific case of Your data.

While preparing the data for CF I would take care about those things:

• dataset size and sparsity - there are many CF algorithms and they are giving different results depending on dataset size (users and items) and its data density. Simple intuition: memory-based KNN methods will be suffering from data sparsity while on the same setup the MF can perform better. Good comparison is presented in this article: A Comparative Study of Collaborative Filtering Algorithms. Moreover, CF methods can depends on the number of users and items differently. To reduce sparsity, You removing items/users which have lower number of ratings/click than selected threshold. This is iterative process. As it's said in the question, when You remove particular item/user it can happen that different user/item respectively can be below the threshold. For research purposes, I even have prepared the function which calculate items/user threshold in order to receive the given sparsity.
• outliers in the meaning of filtering unusual users and items. Those, which can influence Your model greatly, but in a wrong way. In my cases (e-commerce website) it were cases like: really heavy users - other shops, bots (scrapping scripts) that was collecting data, internal traffic (blacklist IPs), bidding automation tools, funny offers which lands on a digg-like webpages. Someone can say that this can be removed by user/item bias. Yes, as soon as Your CF method is using it :-)
• correct slicing of the dataset. For some reasons (e.g. computational) You have to divide You data on smaller parts. And this is reasonable. Divide and conquer approach. But in such cases before doing this, check if You are not loosing a lot of the information by such slicing. Here is the example: we are slicing offers by categories, categories are formed into a category tree, for this reason the same leaf can be part of few main categories. Thus, slicing by main categories can result in having only a part of the whole user-item interactions about viewed/bought together items in this leaf category.

Those are only few tips from my experience. But I hope this would help.

I would appreciate any edits and sharing the results :-)