Spark ALS: recommending for new users

The question

How do I predict the rating for a new user in an ALS model trained in Spark? (New = not seen during training time)

The problem

I'm following the official Spark ALS tutorial here:

http://ampcamp.berkeley.edu/big-data-mini-course/movie-recommendation-with-mllib.html

I'm able to build a good recommender with a decent MSE but I'm struggling with how to input new data to the model. The tutorial changes the first user's ratings prior to training, but this is really a hack. They give the following hint:

9.2. Augmenting matrix factors:

In this tutorial, we add your ratings to the training set. A better way to get the recommendations for you is training a matrix factorization model first and then augmenting the model using your ratings. If this sounds interesting to you, you can take a look at the implementation of MatrixFactorizationModel and see how to update the model for new users and new movies.

The implementation does not help me at all though. Ideally, I'm looking for something like:

predictions = model.predictAllNew(newinput)

But not such method exists. I could go and modify the original RDD, but I think that would require me to retrain the model, so that wouldn't be an ideal solution either. Surely there must be a more elegant way?

Where I am right now:

I think I need to find the latent representation of the new vector. According to the original paper we can compute this like so:

$X_u = (Y^T C^u Y + \lambda I)^{-1} Y^T C^u p(u)$

But when I calculate using the values in the paper, it doesn't match the values from the model. I fix alpha and the regularization parameter, but I think the MLLIB implentation has a different $C^u$ implementation. It's defined here (see line 1304), but not being adept at Scala, this is very hard to reverse engineer for me ...

My current attempt:

V = model.productFeatures().map(lambda x: (x[1])).collect() #product latent matrix Y

Cui =  alpha * np.abs(newinput)
Cui =  (1. + Cui) / (Cui)
Cui[np.where(newinput == 0)] = 0
Cui = np.diag(Cui)

lambdaI = len(np.where(newinput!=0)) * regularization_parameter * np.eye(np.shape(V)[1]) #
term   = np.dot(np.dot(Vt,Cui),V)+lambdaI
term   = np.dot(np.linalg.inv(term),Vt)
term   = np.dot(term,Cui)
term   = np.dot(term,newinput)
latentinput = term


But this doesn't match.

Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide some default recommendations.

Of course, once you have any data, and you can rebuild the model to incorporate the user, you can make recommendations. You can do that in Spark but you already know that. This will take too long if you need to add information about new users at runtime. The technique you want is called "fold-in", used to figure out what the new user vector is (approximately) given the items the user interacts with. It's just some linear algebra and does follow from the equation you give.

I dug out an old slide which might help:

The "Cu" is not really different. I added an 'extension' to handle the case of negative input, but it's the same for positive input.

Here's an implementation of fold-in, although I think it's going to be too dense to be of much value:

https://github.com/OryxProject/oryx/blob/2c01d496f93f2825a076eb7fe492aa39a5290aa6/app/oryx-app-common/src/main/java/com/cloudera/oryx/app/als/ALSUtils.java#L74

Computing the new user vector implied by a user-item interaction is fairly easy linear algebra. The tricky part I've found is deciding how much to weight it.

Hope that is a push in the right direction.