1
$\begingroup$

I used Spark's ALS implementation of matrix factorization (Collaborative Filtering for Implicit Feedback) to train user and item embeddings.

Since we have a lot of users in system, I had to sample some users to train model to avoid overfitting.

Now how do I construct user embeddings for out of training users. I tried constructing user embeddings by averaging item embeddings for user's items. But when I compared performance of average vector vs original user embeddings, it is not that great.

So how would I generate user embeddings using item matrix and rating matrix?

Improve this question
$\endgroup$
3
  • $\begingroup$ Why do you sample users?. You wont overfit the model using more data... $\endgroup$
    – lsmor
    Apr 8, 2019 at 11:22
  • $\begingroup$ We have more than 15 million users, so if I add all them spark ALS training, it crashes with out of memory error. So sampling is the best work around. $\endgroup$
    – Sn.
    Apr 9, 2019 at 0:04
  • $\begingroup$ What you are facing is the cold start problem. Essentialy there is no way to recommend products for new users besides using some default recommendations. There is something call fold-in but I am not sure if it works good. You can split your data in chunks, train the model for the first chunk, save it, and then retrain the model for the second chunk and so on. This way you don't need the whole data set in memory but training will take longer $\endgroup$
    – lsmor
    Apr 9, 2019 at 8:48

1 Answer 1

Reset to default
0
$\begingroup$

Making predictions for new users is often called the cold start problem.

This problem is difficult to over come in a purely item-to-item recommendation system. The most common solution is to include more than just item interaction information. Examples include user profile information and item content information.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.