R Recommender System for very sparse matrices required

Question

I am trying to build a recommender system based on a large and very sparse matrix. Dimensions of that matrix would approximately be 12000 x 37000, possibly even more rows up to 100000. However, this matrix is extremely sparse. With the 12000x37000 version, about 0.053% of the matrix is non-NA. I've tried SVD, but alas, to no avail. To ensure I have not caused any error during my proceedings:

1. I created a data.table with unique triplets of "User" - "Item" - "Rating". I should mention "Rating" can stretch anywhere from 0 to about 150.

2. Now, I applied dcast.data.table to the triplet table, corrected the problem with the first value becoming a column, converted to a matrix. Now, I had a matrix with users as rows, items as columns and the rating as cell content.

Split into Test and Validation set, replaced "NA" with 0, subtracted row means for every row, applied propack.svd from the "svd" package to that matrix, multiplied the three matrices delivered by propack and added the row means to it. (user-means).

After that, I compared the values from the validation set to the corresponding values in my prediction matrix.. and no surprise, the Root mean square error was horribly high, around 6-7.(Mean of non-NA values is around 4.5). I've tried multiple variants of normalisation too, but I just could not get the RMSE below 5.8, ever.

Is there any way to build a viable item recommender system for this dataset? Possibly via arules or clustering?

Answer 1

I would look into the Soft Imputation method that has an implementation in R. It uses iterative soft-thresholding to compute missing values. Calculations are done with a matrix class called "Incomplete" to deal large sparse matrices and allows for quick calculation of scaling/centering rows and columns.

I've had good success using this completing a 10,000 by 10,000 very sparse matrix so I'd imagine it should do fairly well with your dataset.