# AUC ROC Threshold Setting in heavy imbalance

I am doing binary logistic regression on a dataset with very heavy class imbalance. Class 1 is only 1% of data. When I train logistic regressor without class weights I get ROC AUC Score of 0.6269. Which is decent. However, when I see my confusion matrix I see that my model never predicted any 1's at all. So why is my AUC so high? I though AUC is meant to be a good measure for such a case.

Confusion matrix
Predicted      0    All
True
0          32109  32109
1           1223   1223
All        33332  33332


I know Confusion matrix makes the probability threshold 0.5, so is score saying there is some threshold for which model will give higher recall? How can I get this threshold?

      Class  precision    recall  f1-score   support

0       0.96      1.00      0.98     32109
1       0.00      0.00      0.00      1223


I only care about precision and recall of class 1.

-->ROC curves should be used when there are roughly equal numbers of observations for each class.

-->Precision-Recall curves should be used when there is a moderate to large class imbalance.

enter code here
from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(y_test, probs)
auc = auc(recall, precision)

• Ok, well in completition I am doing, ROC curves are used. But thats fine, how do I decide the threshold based on Precision-Recall curve for max precision/recall for class 1? Your code gave me auc of 0.054 – Rahul Deora Jul 7 '19 at 9:55
• the threshold should be pos_label=1 in the above code.. – tharun___ data enthusiast Jul 7 '19 at 12:28
• if you want to write your own code then it is up to your requirement as you can specify threshold based on domain – tharun___ data enthusiast Jul 7 '19 at 12:29

Yes, there must be some threshold values that will produce less trivial classifications. In an imbalanced situation like yours, the relevant thresholds may well be fairly small. There will be a tradeoff, so there won't be just one threshold for you to obtain. You could plot the ROC, maybe along with some threshold information to help you find a threshold that produces a point on the ROC curve that optimizes your use case objective.

The PR curve might be more useful for you, but I wouldn't say that the ROC is necessarily worse. https://stats.stackexchange.com/questions/262616/roc-vs-precision-recall-curves-on-imbalanced-dataset