I am working on a binary classification problem with imbalanced data. The dataset consists of 85% 'false' labels and 15% 'true' labels. I have about 1500 data points and am training with Sklearn ensemble tree methods and XGBoost. I know tree methods tend to work better with more data points but I'm not sure what sampling methods would help. I'm trying to optimize for more 'true' points to be in the top decile. I tried to oversample the 'true' class but since I'm using a top decile conversion, the top decile get artificially populated with a lot of 'true' points. This gave inconsistent conversion scores when I used stratified k-fold cross validation.

Should I split the datasets by class and apply sample techniques? What are some relevant techniques that I can try? I'm working with Python and any relevant libraries would be helpful. Thanks in advance!


Tree-based classification models are generally insensitive to oversampling.

That's because when you upsample the 'true' class, it affects leaf impurity metrics (such as Gini or entropy) only marginally. Imagine a leaf with $a$ 'true' and $b$ 'false' examples - the Gini would be $2\frac{ab}{a+b}$. When you upsample 'true' class by factor of $\lambda$, the Gini becomes $2\lambda\frac{ab}{a\lambda+b}$. If $a<<b$ (which is your case), then this change is approximately just multiplying all the Ginis by $\lambda$, which does not affect the choice of a next tree split.

Thus my advice would be to give up upsampling attempts, and concentrate on feature engineering, parameter tuning, and maybe getting more real observations. If you still want resampling, then you can just set unequal class_weights when fitting your trees - it will affect trainging, but keep cross-validation metrics intact.

If you are interested in top 10% conversion, you may well approximate this metric by ROC AUC and optimize the latter. AUC is not sensitive at all to over- and undersampling. If your model is precise enough, then most of the 15% 'true' examples will be somewhere around this top 10%. Thus, ranking performance measured by AUC and selection performance measured by 10% conversion would strongly correlate, and you can select your best model by AUC.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.