7
$\begingroup$

I am using XGBoost for payment fraud detection. The objective is binary classification, and the data is very unbalanced. One out of every 3-4k transactions is fraud.

I would expect the best way to evaluate the results is a Precision-Recall (PR) curve, not a ROC curve, since the data is so unbalanced.

However in the eval_metric options I see only area under the ROC curve (AUC), and there is no PR option. https://github.com/dmlc/xgboost/blob/master/doc/parameter.md

Also the documentation recommends AUC http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html

Does it make sense to not use a Precision-Recall (PR) curve?

$\endgroup$
6
$\begingroup$

David, you can use mean average precision ('map') or even better logloss ('logloss'). Yes, for unbalanced data precision and recall are very important. I would suggest individually examining these metrics after optimizing with whatever eval_metric you choose.Additionally, there is a parameter called scale_pos_weight, which will help tell the model the distribution of you data. I have found this to greatly improve the performance of "rare event" cases. The following markdown doc has a list of all the parameters and their options. https://github.com/dmlc/xgboost/blob/master/doc/parameter.md

| improve this answer | |
$\endgroup$
9
$\begingroup$

I am answering my own question, almost two years later. XGBoost now has a new eval metric aucpr.

https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.