6
$\begingroup$

I have a dataset with 85k positive labels and 53k negative labels. For this use-case, I am trying to maximize my efforts to the negative class (accurately identify true negatives, and minimize false negatives).

Currently, I am able to train a xgboost classifier to 71% accuracy and my confusion matrix looks like this when benched against a test set

[ 3890 | 8887 ]
[  844 | 20044]

Again for this task, I'd really like to improve my recall and minimize false negatives. However, even in the models' current state, if I try to submit 43k new records for prediction (results are unknown), my model predicts that all 43k records are non-compliant.

Given this information, my two questions are as follows:

  1. How irregular is this occurrence. I thought that skewing to the majority class was possible, but not the minority.
  2. Are there any 'best practices' that can be applied to reduce the aggressiveness of my model? not only does it predict that all claims are negative (0), but it does so with very strong confidence (minimum 'probability of occurrence' is above 90%). Again, less than 40% of the records in my trainset are of the minority class, so I don't understand why it's skewing this hard.

Thanks

$\endgroup$
3
  • 6
    $\begingroup$ Hi! As a sanity check, can you take some of your positive training examples and put them through your inference pipeline and confirm they are still coming out positive? Can you find any examples in your 43k which you know should be positive given the training data? Does the new data have any missing values? Is it featurized in the same way as the training set? Without know more I wouldn't worry about the degree of imbalance you have but would focus more on sanity checks to make sure nothing has changed in terms of the model/feature pipelines between training and scoring. $\endgroup$ – Brandon Donehoo Sep 21 '20 at 3:10
  • 2
    $\begingroup$ @BrandonDonehoo Hi to you as well! Thanks for your reply, this is great stuff. If you don't mind, I do have a few questions. In order of the way you posed your recommendations: 1. To clarify, you suggest training with ONLY positive cases, then make predictions on my production set and just see if it will make positive predictions? 2. The data is too sparse for this kind of check to be done manually :(. 3. This is something I should have checked for. I will just drop any rows if a NaN exists and see how it behaves! 4. Yes, predset is being hash encoded. (Considering Target) Thanks again! $\endgroup$ – Nick Bohl Sep 21 '20 at 12:27
  • $\begingroup$ Any results from the testing you alluded to in your last comment? You allude in the post to the model being very confident that all the test samples are negative; is it similarly confident about negative and/or positive samples in the training set? Could you provide the feature preprocessing code? $\endgroup$ – Ben Reiniger Dec 31 '20 at 15:48
1
$\begingroup$

First of all, your confusion matrix looks normal to me. As shown below, the majority is positive class, and the model prefers to predict most of the negative cases as positive, which is common in imbalanced classes. Your model predicted 8887 negative cases as positive.

True   0 [ 3890 | 8887 ]
Labels 1 [  844 | 20044]
            0       1   
         predicted labels

Your classes are imbalance, so the model does the fit/prediction in favor of majority class.

To overcome this issue, you must perform over/under-sampling to eliminate possible issues caused by imbalance classes. You can use imbalanced-learn library in this regard.

So to answer your questions:

A1: You read the confusion matrix incorrectly. The model is skewing to the majority class.

A2: You must balance your data. Don't forget to perform over/under-sampling on your training set only. In addition, you may need to use precision or recall but not accuracy only, based on your need; for example if one class is important and having less false positive or less false negative is your goal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.