3
$\begingroup$

I have to build a recommender system & it will be evaluated using map@10 criteria. I have rolled up the data/rows at user-item level & is using Gradient Boosting in scikit learn to build the model.

Challenge

But there is a big class imbalance where only 6%( out of 40 K use-item pairs) have actually purchased the item in the entire base. Given this challenge i want to know the accuracy metric I should use to tune the parameters of GBM model.

Question

I thought about using recall, precision, f1 score etc. to tune the hyperparameters of GBM. But which one will maximize map@10 given the class imbalance?

Thanks

Improve this question
$\endgroup$
5
  • $\begingroup$ I guess MAP@10 is mean average precision with test set of one user with from 1 to 10 items to rate, and scored by comparing ranking of those items? So your goals is to correctly rank ratings for a user for some new/unknown items? $\endgroup$
    – Neil Slater
    Sep 2, 2015 at 12:46
  • $\begingroup$ @Neil it's not rating but come up with a list of 10 coupons a user is likely to buy in next one week. To be precise I am working on this problem kaggle.com/c/coupon-purchase-prediction $\endgroup$
    – GeorgeOfTheRF
    Sep 2, 2015 at 13:08
  • $\begingroup$ OK, not ratings but probability then. Metrics are not the same as training objectives (which may be constrained by what the model can optimise directly). If you don't have the metric available already in your library of choice, you could just code it. Often they are only a few lines to implement. In addition, worth checking the forums, one of the first threads or scripts to be created is often an implementation of the metric in R/Python/Matlab etc $\endgroup$
    – Neil Slater
    Sep 2, 2015 at 13:17
  • $\begingroup$ I know how map@K is implemented but I am using a ML classifier which will give probability of a user - item pair to be purchased, then I want to sort the items by this probability for each user and then use ap@k to evaluate the recommender system. So my question is about the metric to evaluate the ml classifier given imbalance of class. I cannot use ap@k to tune parameters of classifier which gives probability for a single user - item right? $\endgroup$
    – GeorgeOfTheRF
    Sep 2, 2015 at 13:27
  • $\begingroup$ Yes you can use the metric to tune hyper-parameters, to trigger early stopping etc, but you cannot use it as an objective function during learning phase. The question is then only slightly different - which objective function is going to be most compatible with the given metric. Which I don't know, unfortunately $\endgroup$
    – Neil Slater
    Sep 2, 2015 at 14:32

1 Answer 1

Reset to default
3
$\begingroup$

If you want to stay in Python, I would suggest going for xgboost instead of GradientBoostingClassifier - among other advantages, it supports map / ndcg as metrics. If you are considering R, gbm supports both map and ndcg.

Improve this answer
$\endgroup$
4
  • $\begingroup$ What is map? I have heard about ndcg. Will prediction accuracy be the same between scikit learn gbm and xgboost? $\endgroup$
    – GeorgeOfTheRF
    Sep 2, 2015 at 13:09
  • $\begingroup$ map = mean average precision, i.e. your MAP@10. I think xgboost accuracy might be higher, because it optimizes MAP directly and has more parameters to customize. $\endgroup$
    – kpb
    Sep 2, 2015 at 13:16
  • $\begingroup$ Oh you are saying xgboost gives option of using map@k for evaluating a binary classification? Can you add link or some example to the answer please ? $\endgroup$
    – GeorgeOfTheRF
    Sep 2, 2015 at 13:29
  • $\begingroup$ Here you go: github.com/dmlc/xgboost/blob/master/doc/parameter.md and then check the section "Learning Task Parameters". $\endgroup$
    – kpb
    Sep 2, 2015 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.