3
$\begingroup$

My data has an extreme class imbalance - 99.7% is 0's and 0.2% is 1's and almost all the predictor variables (6 out of 7) are categorical. I trained an xgboost classifier after performing an upsampling (using ROSE in R). Evaluated the model on 20% test data and the following is the result, enter image description here

Recall, as you can see, is not great. Additionally, if I run the model on a completely different test data (meaning not a chunk of the original data), then the recall value drops to exactly 50%. Is this happening because it is overfitting? Can someone please let me know how I can better this model, considering the extreme class imbalance?

I'm a newbie in the field. Any help would be much appreciated. Thanks!

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Some thoughts - (1) Xgboost lets you do class weighting with the scale_pos_weight parameter. You might try that. (2) 64% recall with such unbalanced data doesn't seem too bad, actually. (3) Recall and precision are baed on a set probability cutoff. You could ask Xgboost to return the class probability rather than the classification, and measure your model quality with AUC instead of recall. $\endgroup$
    – tom
    Commented Dec 19, 2019 at 17:45

2 Answers 2

Reset to default
2
$\begingroup$

The drop from 64 to 50 doesn't seem like overfitting, if your test set was properly held back from model development. (And from the confusion matrix it looks like you haven't made the fairly common mistake of upsampling before splitting.)

If the "completely different test data" is not identically distributed to the training data, that can account for the discrepancy. There aren't universally good ways to account for failure of the iid data assumption; you could try to underfit the model more, on the assumption that there's some underlying large trend that's constant across datasets and the model is finding (and overfitting to) some additional, more localized, trend in the train/validation set.

Improve this answer
$\endgroup$
-1
$\begingroup$

Try using SMOTE before applying XGboost.

Follow the details from this link: https://www.analyticsvidhya.com/blog/2016/03/practical-guide-deal-imbalanced-classification-problems/

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi thanks for the reply! I had tried SMOTE earlier and it didn't help. $\endgroup$
    – FMJ
    Commented Jun 18, 2019 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.