Implicit Training Models in Spark MLlib?

Question

I have something of a ground floor question I’d like to ask. I’m looking at various options for recommendation engines using Spark. I feel that I have a decent grasp of the basics of collaborative filtering and have been able to replicate the simple examples provided by the Spark docs and AMPlab. However, I am seeking to investigate how to create an effective Product Recommendation Algorithm using Implicit Feedback where there is an abundance of user level data available. Let’s say that I have a large dataset with individual user attributes such as a unique user ID, Product ID, user demographic data, user behavioral data, etc. What this data set does not contain is an explicit rating that the user gave the product. In the absence of that explicit feedback, is it still possible to train a Recommendation model using:

ALS.trainImplicit(ratings, rank, numIterations, alpha=0.01)

Furthermore, I found this statement from the Spark website to be a bit nebulous: “If the rating matrix is derived from other source of information (i.e., it is inferred from other signals), you can use the trainImplicit method to get better results.” Does this mean that the large table from my hypothetical example would need to be compacted or reduced to a singular value for the ‘rank’ argument in the trainImplicit function? I have looked everywhere I can think of, but cannot find an accessible example describing how this would be accomplished. If anyone knows of a good resource, please let me know. Lastly, if anyone provides a code example, please do so in Python since that is my preferred Spark API.

Thanks in Advance.

Answer 1

Implicit feedback simply means that given a ratings-dataset, the zeroes are treated as "nothing was rated" as opposed to "the user rated this as a zero". The algorithm they use is based on the one in this paper. The code is pretty difficult to read but available here.

That being said, if you do have binary product choice variables (i.e., user A chose product B, but not the actual ratings), the implicit training could also work as is, even if was not originally designed for it.