How can we show that one model might have higher accuracy than another model but at the same time lower AUC?

Question

Assume that we have two classification models M1 and M2 that are evaluated on five test instances. How to show with an example that M1 can have a higher accuracy than M2, while at the same time M2 has a higher area under the ROC curve (AUC) than M1?

Answer 1

Normally this can happen when there's imbalance in your classes. Imagine that you want to predict these 5 values:

y_test = [1,0,0,0,0] 

and your models M1 and M2 predict the following:

M1_pred = [0,0,0,0,0] 
M2_pred = [1,0,0,1,1] 

Clearly we can see that M1 is biased towards class 0, getting 4/5 results correct (80% accuracy!). While the second one, only gets 3/5 values right (60% accuracy).

However, because AUC ROC is defined by both true positive rate against the false positive, combining those the M2 classifier gets a 75% auc, while M1 only a 50% because it has failed the only one positive, thus penalising the final score.

Try running this toy example:

from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_curve, auc
import matplotlib.pyplot as plt

y_test = [1,0,0,0,0] #ground truth

M1_pred = [0,0,0,0,0] #predictions M1
M2_pred = [1,0,0,1,1] #predictions M2

#Calculate both accuracy scores
M1_acc = accuracy_score(y_test, M1_pred) 
M2_acc = accuracy_score(y_test, M2_pred) 

#Calculate both ROC AUC
fpr1, tpr1, _ = roc_curve(y_test, M1_pred)
roc_auc1 = auc(fpr1, tpr1)
fpr2, tpr2, _ = roc_curve(y_test, M2_pred)
roc_auc2 = auc(fpr2, tpr2)

#Plot both graphs
plt.figure(figsize = (12,6))

#Graph M1
plt.subplot(121)
lw = 2
plt.plot(fpr1, tpr1, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)\n ACC Score = %0.2f' % (roc_auc1, M1_acc))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic ROC')
plt.legend(loc="lower right")

#Graph M2
plt.subplot(122)
plt.plot(fpr2, tpr2, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)\n ACC Score = %0.2f' % (roc_auc2, M2_acc))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic ROC')
plt.legend(loc="lower right")

plt.show()

and you will get something like this: