2
$\begingroup$

I am training a binary classifier in a dataset using AUC as a score. The dataset has two main groups (we will refer to them as good and bad population). A property that this dataset has is having a higher proportion of target = 1 in the bad population.

For this reason, a relatively dummy classifier would give higher scores to the bad population and lower scores to the good population. In fact, the AUC of the classifier could be pretty high globally, and, when looking at the AUC inside both populations separately, the AUC might be really low in both of them.

I want to avoid this behavior. In fact, I am willing to sacrifice some AUC in the global population such that the AUC in each group is not very low. An idea that I had was using the harmonic mean of the AUC of both groups as a metric instead of the general AUC. However, this might not really help a classifier in a natural way.

Are there any papers/techniques/software that can help me in solving this problem in a more natural way?

Improve this question
$\endgroup$
4
  • $\begingroup$ How would one even define AUC in each group? Wouldn't the FPR be undefined if only considering the group of positives (as you would only have TP or FN)? Or are those good/bad groups different from the target groups? $\endgroup$
    – oW_
    Commented Oct 7, 2019 at 17:45
  • $\begingroup$ They are different from target groups. As an example, good population might have 10% of target 1 and bad population might have 50% of target 1. Does this answer your question? $\endgroup$
    – David Masip
    Commented Oct 8, 2019 at 8:12
  • $\begingroup$ Would training two different classifiers (one for the bad, one for the good) be an option? $\endgroup$
    – Romain Reboulleau
    Commented Oct 10, 2019 at 3:14
  • $\begingroup$ It is an option, of course. However, I don't think it uses all the data optimally, as none of the samples of the bad population are used to train the good, and vice-versa. I was wondering if there's something between this strategy and training with all the data. $\endgroup$
    – David Masip
    Commented Oct 10, 2019 at 7:38

1 Answer 1

Reset to default
0
$\begingroup$

Given that in your data there is correlation between population type (good vs. bad) and target, your model may learn undesirable associations between both. Therefore, the population type is a confounding factor.

A natural tool to cope with scenarios with confounders it causal inference. You can find an overview of causal inference in Judea Pearl's work, either this article or his book. A terser introduction to causal inference can be found in Ferenc Huszár's blog including an entry for controlling confounders.

There are a few python packages providing causal inference functionality, such a Microsoft's dowhy or Causalinference.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.