XGBoost skews towards minority class

Question

I have a dataset with 85k positive labels and 53k negative labels. For this use-case, I am trying to maximize my efforts to the negative class (accurately identify true negatives, and minimize false negatives).

Currently, I am able to train a xgboost classifier to 71% accuracy and my confusion matrix looks like this when benched against a test set

[ 3890 | 8887 ]
[  844 | 20044]

Again for this task, I'd really like to improve my recall and minimize false negatives. However, even in the models' current state, if I try to submit 43k new records for prediction (results are unknown), my model predicts that all 43k records are non-compliant.

Given this information, my two questions are as follows:

1. How irregular is this occurrence. I thought that skewing to the majority class was possible, but not the minority.
2. Are there any 'best practices' that can be applied to reduce the aggressiveness of my model? not only does it predict that all claims are negative (0), but it does so with very strong confidence (minimum 'probability of occurrence' is above 90%). Again, less than 40% of the records in my trainset are of the minority class, so I don't understand why it's skewing this hard.

Thanks

Answer 1

First of all, your confusion matrix looks normal to me. As shown below, the majority is positive class, and the model prefers to predict most of the negative cases as positive, which is common in imbalanced classes. Your model predicted 8887 negative cases as positive.

True   0 [ 3890 | 8887 ]
Labels 1 [  844 | 20044]
            0       1   
         predicted labels

Your classes are imbalance, so the model does the fit/prediction in favor of majority class.

To overcome this issue, you must perform over/under-sampling to eliminate possible issues caused by imbalance classes. You can use imbalanced-learn library in this regard.

So to answer your questions:

A1: You read the confusion matrix incorrectly. The model is skewing to the majority class.

A2: You must balance your data. Don't forget to perform over/under-sampling on your training set only. In addition, you may need to use precision or recall but not accuracy only, based on your need; for example if one class is important and having less false positive or less false negative is your goal.