I am building a small recommender system which aims at recommending ~10 products to customers. Instead of using a multi-label classification model, I have opted to build a separate scoring model for each product allowing me to take the other targeted products as features (e.g, if a customer has product X, that information could help predict detention of product Y ; conversely, knowing that a customer has product Y could help predicting detention of product X).
So I end up with ~10 models (say model X, model Y, etc.) with almost the same dataset (only the target changes, and the feature corresponding to that target is removed). However, as each model is different, I am not sure how to compare the scores I obtain for all my products and how to make the best recommendation.
For example, suppose that for a given customer:
score for product X is 0.7 and score for product Y is 0.8 but the precision of model Y is not as good as model X. Should I really recommend product Y?
I was suggested to standardize the scores for each product across customers to end up on the same scale. However, if a product ends up with very higher scores than others, this step would in effect penalize this product.
PS : the detention of each product is pretty low (between 0.5 % to 2 %). I thought a first step would be to re-balance the dataset for each model (say with under of over sampling) in order to have the same class imbalance for each model as I observed that re-balancing biases the scores upwards (unfortunately without really improving the performance of my models).