# Which accuracy metric of a ML classifier can maximize map@K of a recommender system for an unbalanced dataset?

I have to build a recommender system & it will be evaluated using map@10 criteria. I have rolled up the data/rows at user-item level & is using Gradient Boosting in scikit learn to build the model.

Challenge

But there is a big class imbalance where only 6%( out of 40 K use-item pairs) have actually purchased the item in the entire base. Given this challenge i want to know the accuracy metric I should use to tune the parameters of GBM model.

Question

I thought about using recall, precision, f1 score etc. to tune the hyperparameters of GBM. But which one will maximize map@10 given the class imbalance?

Thanks

• I guess MAP@10 is mean average precision with test set of one user with from 1 to 10 items to rate, and scored by comparing ranking of those items? So your goals is to correctly rank ratings for a user for some new/unknown items? Sep 2 '15 at 12:46
• @Neil it's not rating but come up with a list of 10 coupons a user is likely to buy in next one week. To be precise I am working on this problem kaggle.com/c/coupon-purchase-prediction Sep 2 '15 at 13:08
• OK, not ratings but probability then. Metrics are not the same as training objectives (which may be constrained by what the model can optimise directly). If you don't have the metric available already in your library of choice, you could just code it. Often they are only a few lines to implement. In addition, worth checking the forums, one of the first threads or scripts to be created is often an implementation of the metric in R/Python/Matlab etc Sep 2 '15 at 13:17
• I know how map@K is implemented but I am using a ML classifier which will give probability of a user - item pair to be purchased, then I want to sort the items by this probability for each user and then use ap@k to evaluate the recommender system. So my question is about the metric to evaluate the ml classifier given imbalance of class. I cannot use ap@k to tune parameters of classifier which gives probability for a single user - item right? Sep 2 '15 at 13:27
• Yes you can use the metric to tune hyper-parameters, to trigger early stopping etc, but you cannot use it as an objective function during learning phase. The question is then only slightly different - which objective function is going to be most compatible with the given metric. Which I don't know, unfortunately Sep 2 '15 at 14:32