I've been researching on how to develop a hybrid recommender system for a simple book dataset, the main goal is to use both explicit data (purchases) and latent factors (features) to make the recommendations so I finally ended up choosing LightFM as the best option. I started with Surprise but then I realized that there's no way I can implement a Matrix Factorization model there that uses both item data and user data for the predictions.

So my dataframe looks something like this (simplified):

    number       type   username  product  price        model publishing_dt     author          genres
0        6     access   kerrigan     2345  12.99  printedbook    2020-02-01       john    fantasy,kids
1        4     access   kerrigan      897  14.95  printedbook    2019-03-05      alice         fantasy
2        1  orderline  45michael    86833   2.65        ebook    2020-02-04     joseph      action,war 
3        1  orderline   kerrigan    86833   2.65        ebook    2020-02-04     joseph      action,war
1        1  orderline  45michael      897  14.95  printedbook    2019-03-05      alice         fantasy

Where type is a classification of the access made by the user: if it was an order (orderline) or if it was just a view (access), in the case type = access number indicates the amount of times the user accessed the book, and product is the unique id of the book (I think the other fields are self-explanatory).

I'm using purchased or not purchased as the interaction matrix as I don't have any kind of rating:

product        2345     897    86833    
45michael      0.0      1.0      1.0     
kerrigan       0.0      0.0      1.0  

The main problem I've found with Surprise is that it would only use the values of that matrix to make the predictions, so as 45michael bought 86833 and 897 and Kerrigan also bought 86833, I'm assuming it will predict that Kerrigan will also buy 897 because of the similarity between both users (which is not a mistaken assumption).

But I would like the system to use the latent factors given by author and genres, that's how I ended up on LightFM.

So I would have an item-features matrix combining authors and genres:

genre        john    alice   joseph   fantasy   kids   action   war             
2345          1.0      0.0      0.0    1.0      1.0     0.0     0.0   
897           0.0      1.0      0.0    1.0      0.0     0.0     0.0     
86833         0.0      0.0      1.0    0.0      0.0     1.0     1.0

And an user-features matrix:

genre        john    alice   joseph   fantasy   kids   action   war             
45michael     0.0      1.0      1.0     1.0      0.0    1.0     1.0
kerrigan      0.0      0.0      1.0     0.0      0.0    1.0     1.0

I was trying to follow this article: https://towardsdatascience.com/build-a-machine-learning-recommender-72be2a8f96ed which seemed to cover my fundamental problems but as I have never used LightFM I'm having a difficult time understanding how to define a model to use with this kind of data and the creation of the item-features and user-features matrices seems pretty complicated, can I use a sparse matrix created with scipy as input?

Another doubt that I have (it may seem basic knowledge but I'm considerably new to this machine-learning/recommendation system world) is that I've been splitting the dataset into train and test sets when testing Surprise library but I want to be able to make predictions for all of the users in the dataset, even those used for training, should I make the whole dataset a training set? But what happens when the dataset grows too big for this case?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.