0
$\begingroup$

I am building a small recommender system which aims at recommending ~10 products to customers. Instead of using a multi-label classification model, I have opted to build a separate scoring model for each product allowing me to take the other targeted products as features (e.g, if a customer has product X, that information could help predict detention of product Y ; conversely, knowing that a customer has product Y could help predicting detention of product X).

So I end up with ~10 models (say model X, model Y, etc.) with almost the same dataset (only the target changes, and the feature corresponding to that target is removed). However, as each model is different, I am not sure how to compare the scores I obtain for all my products and how to make the best recommendation.

For example, suppose that for a given customer:

score for product X is 0.7 and score for product Y is 0.8 but the precision of model Y is not as good as model X. Should I really recommend product Y?

I was suggested to standardize the scores for each product across customers to end up on the same scale. However, if a product ends up with very higher scores than others, this step would in effect penalize this product.

Any ideas?

PS : the detention of each product is pretty low (between 0.5 % to 2 %). I thought a first step would be to re-balance the dataset for each model (say with under of over sampling) in order to have the same class imbalance for each model as I observed that re-balancing biases the scores upwards (unfortunately without really improving the performance of my models).

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

You can continue to use machine learning for this model-combination step too. You're describing a form of ensembling. How about:

  1. set aside some purchases as your validation set V, and keep the remaining as training set TRAIN
  2. for TRAIN, use k-fold cross-validation to create your separate models
  3. use TRAIN again with k-fold cross-validation to ensemble the models together
  4. report performance of the ensembled models on V

Ensemble techniques will take into account the predictive quality of each individual model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Are your referring to stacking? $\endgroup$
    – Tanguy
    Nov 14, 2017 at 20:58
  • $\begingroup$ Yes, as well as any ensemble method that's appropriate for your problem. $\endgroup$
    – Dan Jarratt
    Nov 15, 2017 at 17:46
1
$\begingroup$

It would be great if you can find any trends which are similar, as you know a generalized model gives you better result when compared to the model which is built for a specific purpose.

1st question is how many data points do you have?, assuming that you have good amount of data to train for each model in each product then it is fine(you can go ahead with your current process) but in a scenario where you have less data points then, you can find if there is a trend similar is followed in 2 different products(will be outcome of your Exploratory Analysis), then you can combine such data together, by this you are generalizing the model.

As Dan Jarratt has said, I would also agree with him. Ensemble Model would help you in getting better results and with good Accuracy, for better understanding you can go through this Link on Ensemble modelling in R.

Do let me know if you have any questions.

Improve this answer
$\endgroup$
9
  • $\begingroup$ I have got enough data (>100k samples). I understand you are both suggesting to use a multi-label meta-model wich will use the other (stacked) models predictions as features. Am I correct (thought about that solution but was wondering if there were other techniques)? $\endgroup$
    – Tanguy
    Nov 15, 2017 at 15:13
  • $\begingroup$ yeah partially right, it wont be using that as features but it would be using either voting system or average of prediction of all the models and give you the best result. $\endgroup$
    – Toros91
    Nov 15, 2017 at 15:20
  • $\begingroup$ Just to make sure the context is clear: I have ~10 models and each model is predicting a different target (every target being the detention of one of the ~10 products). So the models can't directly be averaged as they do not deal with the same target. My goal is to identify, for every customer, which product/model is the most relevant. So if I would be using stacking, I would have ~10 binary sub-models (each model predicting a different target), and a multi-label meta-model wich would predict the ownership of the ~10 products. $\endgroup$
    – Tanguy
    Nov 15, 2017 at 18:35
  • $\begingroup$ @Tanguy: yes you are almost right, is there chance for you to show some sample data? one more question what is your target variable for each model? P.S: sorry for not replying back early, by the time I read your comment it was very late here. $\endgroup$
    – Toros91
    Nov 16, 2017 at 1:06
  • $\begingroup$ Unfortunately the data is confidential. To give you a better idea, I am aiming at recommending the most suitable product from a "basket" of 10 different products. The dataset is unique (100k samples, around 200 features). A binary classification model was built to predict the detention of each product, so I end up with a single score for each product, and 10 scores for every customer (one score for every product). The proportion of detention of each product is generally very low, around 1% (except for two products for which detention reaches 10 % and 20 %). $\endgroup$
    – Tanguy
    Nov 16, 2017 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.