Why ROC value area under curve of two models is different whereas accuracy, precision, recall, f1-score and confusion matrix is same

Question

I am applying logistic regression and support vector machines on the extacly same dataset with 70% data for training and 30% for test. Both perform exactly the same have the same precision, recall, f1-score and confusion matrix. The confusion matrix and classification report for both can be seen below.

       [1180    0]
       [  17    0]]

         precision    recall  f1-score   support

   0       0.99      1.00      0.99      1180
   1       0.00      0.00      0.00        17

   accuracy                               0.99      1197
   macro avg          0.49      0.50      0.50      1197
   weighted avg       0.97      0.99      0.98      1197

I applied 10-fold cross validation to both models too both achieve the same mean accracy of 98.57% both failing to achieve any True Positives across all 10 folds, also have same number of TP,TN,FP,FN in each of the 10-fold

However, the problem is when I compute the roc area under curve values they are different as seen below:

Logistic Regression: 0.86
Support Vector Machines: 0.23

These values indicate logistic regression is much better as compared to support vector machines although there confusion matrix and classification report is the same I am really confused as all the other reuslts indicated same performance by both models ?? Can someone please help why both get different values of roc whereas everything else is the same.

Answer 1

Those scorings looks strange for me, but beside that you must remember that F1, accuracy, confusion matrix, etc depends on the chosen threshold, while AUC is threshold-independent (it is an integral over all of the thresholds from 0 to 1).

Your models return some probabilities of being a member of class 1. If you choose to label by '1' only those, which have probability higher than 99% (this is a threshold) then you get a recall at almost 100% and a precision near 0%.

Answer 2

When you compute the ROC, you're varying the decision threshold, while the confusion matrix and those metrics based on it are using a default threshold (probability 0.5 for the logistic regression, and the max-margin boundary of the SVM [which isn't meant to be probabilistic by itself]). So the logistic regression is doing at least something meaningful at lower thresholds, whereas the SVM continues to do badly.

That the logistic regression model makes no positive-class predictions is probably just due to the default threshold of 0.5, whereas the SVM's decision boundary doesn't actually separate any positive-class test points.

For such an imbalanced dataset, sometimes the auROC can be misleading: if many of the majority class are "easy" to identify as such, then the auROC can be quite high, even if the rest of the majority and minority classes are thoroughly scrambled. (Whether this is "right" or not depends on your point of view, and the context.)