I am doing binary logistic regression on a dataset with very heavy class imbalance. Class 1 is only 1% of data. When I train logistic regressor without class weights I get ROC AUC Score of 0.6269. Which is decent. However, when I see my confusion matrix I see that my model never predicted any 1's at all. So why is my AUC so high? I though AUC is meant to be a good measure for such a case.

Confusion matrix
 Predicted      0    All
0          32109  32109
1           1223   1223
All        33332  33332

I know Confusion matrix makes the probability threshold 0.5, so is score saying there is some threshold for which model will give higher recall? How can I get this threshold?

      Class  precision    recall  f1-score   support

       0       0.96      1.00      0.98     32109
       1       0.00      0.00      0.00      1223

I only care about precision and recall of class 1.


-->ROC curves should be used when there are roughly equal numbers of observations for each class.

-->Precision-Recall curves should be used when there is a moderate to large class imbalance.

enter code here
 from sklearn.metrics import precision_recall_curve
 precision, recall, thresholds = precision_recall_curve(y_test, probs)
 auc = auc(recall, precision)
  • $\begingroup$ Ok, well in completition I am doing, ROC curves are used. But thats fine, how do I decide the threshold based on Precision-Recall curve for max precision/recall for class 1? Your code gave me auc of 0.054 $\endgroup$ Jul 7 '19 at 9:55
  • $\begingroup$ the threshold should be pos_label=1 in the above code.. $\endgroup$
    – findtharun
    Jul 7 '19 at 12:28
  • $\begingroup$ if you want to write your own code then it is up to your requirement as you can specify threshold based on domain $\endgroup$
    – findtharun
    Jul 7 '19 at 12:29

Yes, there must be some threshold values that will produce less trivial classifications. In an imbalanced situation like yours, the relevant thresholds may well be fairly small. There will be a tradeoff, so there won't be just one threshold for you to obtain. You could plot the ROC, maybe along with some threshold information to help you find a threshold that produces a point on the ROC curve that optimizes your use case objective.

The PR curve might be more useful for you, but I wouldn't say that the ROC is necessarily worse. https://stats.stackexchange.com/questions/262616/roc-vs-precision-recall-curves-on-imbalanced-dataset


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.