In XGBoost would we evaluate results with a Precision Recall curve vs ROC?

Question

I am using XGBoost for payment fraud detection. The objective is binary classification, and the data is very unbalanced. One out of every 3-4k transactions is fraud.

I would expect the best way to evaluate the results is a Precision-Recall (PR) curve, not a ROC curve, since the data is so unbalanced.

However in the eval_metric options I see only area under the ROC curve (AUC), and there is no PR option. https://github.com/dmlc/xgboost/blob/master/doc/parameter.md

Also the documentation recommends AUC http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html

Does it make sense to not use a Precision-Recall (PR) curve?

Answer 1

David, you can use mean average precision ('map') or even better logloss ('logloss'). Yes, for unbalanced data precision and recall are very important. I would suggest individually examining these metrics after optimizing with whatever eval_metric you choose.Additionally, there is a parameter called scale_pos_weight, which will help tell the model the distribution of you data. I have found this to greatly improve the performance of "rare event" cases. The following markdown doc has a list of all the parameters and their options. https://github.com/dmlc/xgboost/blob/master/doc/parameter.md

Answer 2

I am answering my own question, almost two years later. XGBoost now has a new eval metric aucpr.

https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters