AUC ROC Threshold Setting in heavy imbalance

Question

I am doing binary logistic regression on a dataset with very heavy class imbalance. Class 1 is only 1% of data. When I train logistic regressor without class weights I get ROC AUC Score of 0.6269. Which is decent. However, when I see my confusion matrix I see that my model never predicted any 1's at all. So why is my AUC so high? I though AUC is meant to be a good measure for such a case.

Confusion matrix
 Predicted      0    All
True                   
0          32109  32109
1           1223   1223
All        33332  33332

I know Confusion matrix makes the probability threshold 0.5, so is score saying there is some threshold for which model will give higher recall? How can I get this threshold?

      Class  precision    recall  f1-score   support

       0       0.96      1.00      0.98     32109
       1       0.00      0.00      0.00      1223

I only care about precision and recall of class 1.

Answer 1

Yes, there must be some threshold values that will produce less trivial classifications. In an imbalanced situation like yours, the relevant thresholds may well be fairly small. There will be a tradeoff, so there won't be just one threshold for you to obtain. You could plot the ROC, maybe along with some threshold information to help you find a threshold that produces a point on the ROC curve that optimizes your use case objective.

The PR curve might be more useful for you, but I wouldn't say that the ROC is necessarily worse. https://stats.stackexchange.com/questions/262616/roc-vs-precision-recall-curves-on-imbalanced-dataset

Answer 2

-->ROC curves should be used when there are roughly equal numbers of observations for each class.

-->Precision-Recall curves should be used when there is a moderate to large class imbalance.

enter code here
 from sklearn.metrics import precision_recall_curve
 precision, recall, thresholds = precision_recall_curve(y_test, probs)
 auc = auc(recall, precision)