2
$\begingroup$

I'm fairly new to data science and machine learning and have been trying to read a bit more on methods like boosting for one of the projects I am working on. The investigator on this project is interested in utilizing xgboost for this project, but there are a couple of main issues that we anticipate once we get the data.

There will be class imbalance, as in way fewer 1's than 0's. I've seen that people handle this by using the scale_pos_weight parameter in xgboost, but from my understanding, xgboost allows for custom objective functions. I was wondering how people come up with custom objective functions and the basis behind their chosen objective function. I'm interested in a customized objective function that penalizes false negatives more and was wondering if there's anything out there on the web that's recommended. We are dealing with a classification problem, so by default we would be using a log-loss function. I am hoping there is some sort of custom log-loss function out there that is commonly used.

I apologize if I am using the wrong terminology, as I'm still very new to this. Thanks so much!

$\endgroup$

2 Answers 2

2
$\begingroup$

The scale_pos_weight will be specifying the tradeoff in the loss function for specifying punishing FPs / TPs more.

https://xgboost.readthedocs.io/en/latest/parameter.html: "Control the balance of positive and negative weights, useful for unbalanced classes. A typical value to consider: sum(negative instances) / sum(positive instances)."

$\endgroup$
2
  • $\begingroup$ Thank you! I was wondering if there's any documentation on what happens under the hood when we use scale_pos_weight. I'm trying to grasp the concepts before actually applying them. $\endgroup$
    – corkee
    Oct 15, 2019 at 17:05
  • $\begingroup$ It’s just weighting either side in the cross entropy loss function $\endgroup$ Oct 19, 2019 at 4:29
1
$\begingroup$

XGBoost has several parameters to tune for imbalance datasets. You wouldn't mess with the objective function from my knowledge. You can find them below:

  • scale_pos_weight
  • max_delta_step
  • min_child_weight

Another thing to consider is to resample the dataset. We talk about Undersampling, Oversampling and Ensemble sampling. I think I was using the imbalanced-learn Python library for that. Things can get even more creative there. For instance, create many XGBoost trees on the same undersampled class instances but each XGBoost tree holds another copy (or randomly sampled copy) of your oversampled class instances. Then you can average your results.

In the end, you will have to try and see for yourself.

$\endgroup$
1
  • $\begingroup$ Thank you! I will take a look. It seems like Python has more tools for imbalance than R, though I'm not sure. $\endgroup$
    – corkee
    Oct 15, 2019 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.