The question
How do I predict the rating for a new user in an ALS model trained in Spark? (New = not seen during training time)
The problem
I'm following the official Spark ALS tutorial here:
http://ampcamp.berkeley.edu/big-data-mini-course/movie-recommendation-with-mllib.html
I'm able to build a good recommender with a decent MSE but I'm struggling with how to input new data to the model. The tutorial changes the first user's ratings prior to training, but this is really a hack. They give the following hint:
9.2. Augmenting matrix factors:
In this tutorial, we add your ratings to the training set. A better way to get the recommendations for you is training a matrix factorization model first and then augmenting the model using your ratings. If this sounds interesting to you, you can take a look at the implementation of MatrixFactorizationModel and see how to update the model for new users and new movies.
The implementation does not help me at all though. Ideally, I'm looking for something like:
predictions = model.predictAllNew(newinput)
But not such method exists. I could go and modify the original RDD, but I think that would require me to retrain the model, so that wouldn't be an ideal solution either. Surely there must be a more elegant way?
Where I am right now:
I think I need to find the latent representation of the new vector. According to the original paper we can compute this like so:
$X_u = (Y^T C^u Y + \lambda I)^{-1} Y^T C^u p(u)$
But when I calculate using the values in the paper, it doesn't match the values from the model. I fix alpha and the regularization parameter, but I think the MLLIB implentation has a different $C^u$ implementation. It's defined here (see line 1304), but not being adept at Scala, this is very hard to reverse engineer for me ...
My current attempt:
V = model.productFeatures().map(lambda x: (x[1])).collect() #product latent matrix Y
Cui = alpha * np.abs(newinput)
Cui = (1. + Cui) / (Cui)
Cui[np.where(newinput == 0)] = 0
Cui = np.diag(Cui)
lambdaI = len(np.where(newinput!=0)) * regularization_parameter * np.eye(np.shape(V)[1]) #
term = np.dot(np.dot(Vt,Cui),V)+lambdaI
term = np.dot(np.linalg.inv(term),Vt)
term = np.dot(term,Cui)
term = np.dot(term,newinput)
latentinput = term
But this doesn't match.
