XGBoost change loss function

Question

I'm using XGBoost (through the sklearn API) and I'm trying to do a binary classification.

False Positives are much worse for me than False Negatives, how can I take this into account?

The API confuses me a bit and I found two arguments that might be related to this:

objective & eval_metric

Which one is it? And how to use it?

Answer 1

In a scenario where consequences of prediction errors are not equivalent, you are usually still interested in training a model to predict accurately from the data set, and would not change the objective function in supervised learning.

Typically when consequences of FP and FN differ, you would:

• Use the confidence of the prediction given by the model. In XGBoost, that is objective: "binary:logistic"

• Use area under ROC as a base metric for deciding best tuned model (this is eval_metric: "auc" for XGBoost). This metric is a measure of how well sorted your classes are - the higher the value, the easier and more effective it is to tune the confidence level later.

• Decide a weighting for your costs FP and FN - this is entirely up to you, you say that FP is worse than FN, but in order to make an optimal decision you have to turn that into a numerical statement. If you can assign a relative financial cost to a business for each kind of mistake, then that would be a good start.

• Using the best tuned model, use the weightings to calculate costs at different cutoffs of class confidence before assigning positive class that you wish to act upon in some way. Predict against a test set to get class probabilities. Then use different cutoff points for positive class, count your FP and FN given that cutoff, and multiply the totals by the costs you chose later. The lowest-scoring cutoff point should be the one to use in production. If FP has a higher cost than FN, then you will likely find that a confidence level > 0.5 is required.

You can also look into machine learning approaches that consider the consequences of actions - e.g. reinforcement learning - in order to achieve a similar result within a single learning framework. However, that is not something you can do solely within XGBoost, and as long as your problem remains balancing between FP and FN in a single step prediction, then the above approach should be OK.

Answer 2

Based on this page's description of what objective and eval_metric do, I don't think those are exactly right.

I would try playing with scale_pos_weights in this case. Go to this link and Ctrl+F for "scale_pos_weights".