I am trying to build an SVD-based recommender system. According to my understanding, the training data should only contain the users who buy at least m items and the items which are bought by n unique users. Otherwise, if I use all users and items (including low-frequency users and items), I think the training data will be noisy.

However, here is the problem: To build the training data for SVD, I first retrieve all users who buy at least m items from the database. But there are some items bought by these users are not bought by at least n users. But after further filtering out these low-frequency items from the training data, some of the remaining users will not have item sets which contain m items because some items are deleted.

I feel that I am not on the right track. How am I supposed to build the training data for SVD?

  • 1
    $\begingroup$ Try using $L_p$ regularization instead of doing that to fight overfitting. $\endgroup$
    – Emre
    Commented Apr 23, 2016 at 6:38

1 Answer 1


I consider this answer as a general approach for the data preparation for CF algorithms and based on my experience. I do not know the specific case of Your data.

While preparing the data for CF I would take care about those things:

  • dataset size and sparsity - there are many CF algorithms and they are giving different results depending on dataset size (users and items) and its data density. Simple intuition: memory-based KNN methods will be suffering from data sparsity while on the same setup the MF can perform better. Good comparison is presented in this article: A Comparative Study of Collaborative Filtering Algorithms. Moreover, CF methods can depends on the number of users and items differently. To reduce sparsity, You removing items/users which have lower number of ratings/click than selected threshold. This is iterative process. As it's said in the question, when You remove particular item/user it can happen that different user/item respectively can be below the threshold. For research purposes, I even have prepared the function which calculate items/user threshold in order to receive the given sparsity.
  • outliers in the meaning of filtering unusual users and items. Those, which can influence Your model greatly, but in a wrong way. In my cases (e-commerce website) it were cases like: really heavy users - other shops, bots (scrapping scripts) that was collecting data, internal traffic (blacklist IPs), bidding automation tools, funny offers which lands on a digg-like webpages. Someone can say that this can be removed by user/item bias. Yes, as soon as Your CF method is using it :-)
  • correct slicing of the dataset. For some reasons (e.g. computational) You have to divide You data on smaller parts. And this is reasonable. Divide and conquer approach. But in such cases before doing this, check if You are not loosing a lot of the information by such slicing. Here is the example: we are slicing offers by categories, categories are formed into a category tree, for this reason the same leaf can be part of few main categories. Thus, slicing by main categories can result in having only a part of the whole user-item interactions about viewed/bought together items in this leaf category.

Those are only few tips from my experience. But I hope this would help.

I would appreciate any edits and sharing the results :-)

  • $\begingroup$ Thanks for your reply! Currently, the test errors of my SVD in most cases are not much better than those of linear regression or average-based deterministic baseline. My guess is that the training data is still sparse although the training data contain frequent users and items. In other words, I concern users do not share enough common items. So I wonder how you calculated sparsity, the fraction of known cells in the user-item matrix? What's the sparsity threshold you often use? $\endgroup$
    – Munichong
    Commented Apr 23, 2016 at 18:40
  • $\begingroup$ Actually, the fraction of known cells in the user-item matrix to all of it will be density. Sparsity is the opposite (100% - density) :-). What is curious in Your case, that You should expect that MF method should perform better when sparsity is raising. For threshold - see de article in the answer. I think the good starting point for everyone starting with CF algos. $\endgroup$ Commented May 4, 2016 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.