2
$\begingroup$

I used Spark's ALS implementation of matrix factorization (Collaborative Filtering for Implicit Feedback) to train user and item embeddings.

Since we have a lot of users in system, I had to sample some users to train model to avoid overfitting.

Now how do I construct user embeddings for out of training users. I tried constructing user embeddings by averaging item embeddings for user's items. But when I compared performance of average vector vs original user embeddings, it is not that great.

So how would I generate user embeddings using item matrix and rating matrix?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Why do you sample users?. You wont overfit the model using more data... $\endgroup$
    – lsmor
    Apr 8, 2019 at 11:22
  • $\begingroup$ We have more than 15 million users, so if I add all them spark ALS training, it crashes with out of memory error. So sampling is the best work around. $\endgroup$
    – Sn.
    Apr 9, 2019 at 0:04
  • $\begingroup$ What you are facing is the cold start problem. Essentialy there is no way to recommend products for new users besides using some default recommendations. There is something call fold-in but I am not sure if it works good. You can split your data in chunks, train the model for the first chunk, save it, and then retrain the model for the second chunk and so on. This way you don't need the whole data set in memory but training will take longer $\endgroup$
    – lsmor
    Apr 9, 2019 at 8:48
  • $\begingroup$ "We have more than 15 million users [..] it crashes with out of memory error". It looks like you are asking a wrong question. You should ask "what recommender algorithm should I use so that my training does not crash with 15 million users". $\endgroup$
    – Valentas
    Jul 10 at 13:49
  • $\begingroup$ Also: you can scale to this many users, you probably don't have enough parallelism $\endgroup$
    – Sean Owen
    Jul 11 at 1:26

1 Answer 1

Reset to default
0
$\begingroup$

Making predictions for new users is often called the cold start problem.

This problem is difficult to over come in a purely item-to-item recommendation system. The most common solution is to include more than just item interaction information. Examples include user profile information and item content information.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.