# Overall AUC higher than all "stratified" AUCs

For one of my binary classification models, I have observed this (Simpson's Rule-esque) paradox. The AUC on the test set as a whole is 0.8.

Gender is one of the model's features. So I decided to produce a "bias" report, for which I calculated AUCs for each of the Male and Female subgroups. But I noticed that each of these AUCs is around 0.7. How is this possible, given that the overall test AUC is 0.8? (In my dataset, every data point belongs to either the Male or the Female subgroup.) I don't expect the overall AUC to simply be a (weighted) linear combination of AUCs for the individual strata.

I'm hoping to get both a technical/mathematical answer and a high-level explanation. Please let me know if any further information is needed (if you think I should plot the overall, Male, and Female ROC curves, for instance). Thank you!

• From @Valentas, take an extreme example. All y=0, males score < 0.25, y=1 males (0.26-0.5), y=0 females (0.51-0.75) and y=1 females (0.76-1). When looking at just males or females the AUROC is 1 since y=0 scores < y=1 scores. Now think what happens when combing the data. Male y=1 scores are < female y=0 scores. This is one of the weaknesses of AUROC. Mar 19 at 9:52
• Thanks, @Craig! But how would you modify this example to show the opposite: a high overall AUC but relatively low AUCs for each of the Male and Female subgroups? Appreciate your help. Mar 20 at 1:58
• Plot your data. Scores on the X-axis. Males y=0 and males y=1 as separate colors and/or shapes. Now you can see what is rank-ordered correctly vs incorrectly. Again with females on a separate plot. Now combined. You can visually see. Sample and jitter as needed. Make sure AUROC is the right metric to use for your business problem. I am not always interested in rank-order power so AUROC is not always applicable. To understand more about AUROC - stats.stackexchange.com/questions/145566/…). Mar 21 at 10:43

AUC can be defined as $$P(X_1 > X_0)$$ where $$X_1$$ is the score of a randomly chosen positive instance and $$X_0$$ is the score of a randomly chosen negative instance.