Class Imbalance and Cost-Sensitive Learning XGBoost

Question

I'm fairly new to data science and machine learning and have been trying to read a bit more on methods like boosting for one of the projects I am working on. The investigator on this project is interested in utilizing xgboost for this project, but there are a couple of main issues that we anticipate once we get the data.

There will be class imbalance, as in way fewer 1's than 0's. I've seen that people handle this by using the scale_pos_weight parameter in xgboost, but from my understanding, xgboost allows for custom objective functions. I was wondering how people come up with custom objective functions and the basis behind their chosen objective function. I'm interested in a customized objective function that penalizes false negatives more and was wondering if there's anything out there on the web that's recommended. We are dealing with a classification problem, so by default we would be using a log-loss function. I am hoping there is some sort of custom log-loss function out there that is commonly used.

I apologize if I am using the wrong terminology, as I'm still very new to this. Thanks so much!

Answer 1

The scale_pos_weight will be specifying the tradeoff in the loss function for specifying punishing FPs / TPs more.

https://xgboost.readthedocs.io/en/latest/parameter.html: "Control the balance of positive and negative weights, useful for unbalanced classes. A typical value to consider: sum(negative instances) / sum(positive instances)."

Answer 2

XGBoost has several parameters to tune for imbalance datasets. You wouldn't mess with the objective function from my knowledge. You can find them below:

• scale_pos_weight
• max_delta_step
• min_child_weight

Another thing to consider is to resample the dataset. We talk about Undersampling, Oversampling and Ensemble sampling. I think I was using the imbalanced-learn Python library for that. Things can get even more creative there. For instance, create many XGBoost trees on the same undersampled class instances but each XGBoost tree holds another copy (or randomly sampled copy) of your oversampled class instances. Then you can average your results.

In the end, you will have to try and see for yourself.