I am working on a real-time recommender system predicting a product to a user using deep learning techniques (like wide & deep learning, deep & cross-network etc). Product catalogue can be huge (1000s to 1 million) and for a given user, the model needs to be evaluated against each product in real-time. As scalability is an important concern, is there any way to reduce the serving time complexity by tuning model architecture?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
You write ... the model needs to be evaluated against each product in real-time., which gets me thinking that you use a binary classification (sigmoid in the final layer) architecture with negative sampling for the user/item interactions when training your model.
Have you considered using multi-class classification instead? Thus, for the user only predict once for the entire product catalogue, and selecting the top-k candidates from the softmax-layer. This way, you only need to feed-forward once through your neural net during inference.
-
1$\begingroup$ the network predicts click-through rate (CTR) for each product and we choose the top CTR products to show to the user. Product_id is an input to the network which makes multiple evaluations against multiple product_ids during inference. I am not sure whether softmax can handle such a scenario. $\endgroup$ Jan 12, 2021 at 13:25
-
$\begingroup$ If "the network predicts click-through rate (CTR) for each product...", then you only need to feed-forward once during inference right? Is this too slow for real-time you mean? $\endgroup$– MarcusJan 12, 2021 at 16:41
-
$\begingroup$ it is a feed-forward, but for a given request data, CTR against each item needs to be computed. if there are 10K products, I would need to prepare a matrix of 10K rows with same request data changing product ids and response needs to be in less than 100 milliseconds. $\endgroup$ Jan 12, 2021 at 19:33