I am training a binary classifier in a dataset using AUC as a score. The dataset has two main groups (we will refer to them as good and bad population). A property that this dataset has is having a higher proportion of target = 1 in the bad population.

For this reason, a relatively dummy classifier would give higher scores to the bad population and lower scores to the good population. In fact, the AUC of the classifier could be pretty high globally, and, when looking at the AUC inside both populations separately, the AUC might be really low in both of them.

I want to avoid this behavior. In fact, I am willing to sacrifice some AUC in the global population such that the AUC in each group is not very low. An idea that I had was using the harmonic mean of the AUC of both groups as a metric instead of the general AUC. However, this might not really help a classifier in a natural way.

Are there any papers/techniques/software that can help me in solving this problem in a more natural way?

  • $\begingroup$ How would one even define AUC in each group? Wouldn't the FPR be undefined if only considering the group of positives (as you would only have TP or FN)? Or are those good/bad groups different from the target groups? $\endgroup$
    – oW_
    Oct 7, 2019 at 17:45
  • $\begingroup$ They are different from target groups. As an example, good population might have 10% of target 1 and bad population might have 50% of target 1. Does this answer your question? $\endgroup$ Oct 8, 2019 at 8:12
  • $\begingroup$ Makes sense. Thanks $\endgroup$
    – oW_
    Oct 8, 2019 at 14:30
  • $\begingroup$ Would training two different classifiers (one for the bad, one for the good) be an option? $\endgroup$ Oct 10, 2019 at 3:14
  • $\begingroup$ It is an option, of course. However, I don't think it uses all the data optimally, as none of the samples of the bad population are used to train the good, and vice-versa. I was wondering if there's something between this strategy and training with all the data. $\endgroup$ Oct 10, 2019 at 7:38

1 Answer 1


Given that in your data there is correlation between population type (good vs. bad) and target, your model may learn undesirable associations between both. Therefore, the population type is a confounding factor.

A natural tool to cope with scenarios with confounders it causal inference. You can find an overview of causal inference in Judea Pearl's work, either this article or his book. A terser introduction to causal inference can be found in Ferenc Huszár's blog including an entry for controlling confounders.

There are a few python packages providing causal inference functionality, such a Microsoft's dowhy or Causalinference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.