6
$\begingroup$

I have a dataset with 85k positive labels and 53k negative labels. For this use-case, I am trying to maximize my efforts to the negative class (accurately identify true negatives, and minimize false negatives).

Currently, I am able to train a xgboost classifier to 71% accuracy and my confusion matrix looks like this when benched against a test set

[ 3890 | 8887 ]
[  844 | 20044]

Again for this task, I'd really like to improve my recall and minimize false negatives. However, even in the models' current state, if I try to submit 43k new records for prediction (results are unknown), my model predicts that all 43k records are non-compliant.

Given this information, my two questions are as follows:

  1. How irregular is this occurrence. I thought that skewing to the majority class was possible, but not the minority.
  2. Are there any 'best practices' that can be applied to reduce the aggressiveness of my model? not only does it predict that all claims are negative (0), but it does so with very strong confidence (minimum 'probability of occurrence' is above 90%). Again, less than 40% of the records in my trainset are of the minority class, so I don't understand why it's skewing this hard.

Thanks

$\endgroup$
  • 5
    $\begingroup$ Hi! As a sanity check, can you take some of your positive training examples and put them through your inference pipeline and confirm they are still coming out positive? Can you find any examples in your 43k which you know should be positive given the training data? Does the new data have any missing values? Is it featurized in the same way as the training set? Without know more I wouldn't worry about the degree of imbalance you have but would focus more on sanity checks to make sure nothing has changed in terms of the model/feature pipelines between training and scoring. $\endgroup$ – Brandon Donehoo Sep 21 '20 at 3:10
  • 2
    $\begingroup$ @BrandonDonehoo Hi to you as well! Thanks for your reply, this is great stuff. If you don't mind, I do have a few questions. In order of the way you posed your recommendations: 1. To clarify, you suggest training with ONLY positive cases, then make predictions on my production set and just see if it will make positive predictions? 2. The data is too sparse for this kind of check to be done manually :(. 3. This is something I should have checked for. I will just drop any rows if a NaN exists and see how it behaves! 4. Yes, predset is being hash encoded. (Considering Target) Thanks again! $\endgroup$ – Nick Bohl Sep 21 '20 at 12:27
  • $\begingroup$ Any results from the testing you alluded to in your last comment? You allude in the post to the model being very confident that all the test samples are negative; is it similarly confident about negative and/or positive samples in the training set? Could you provide the feature preprocessing code? $\endgroup$ – Ben Reiniger Dec 31 '20 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.