Identifying problematic binary features in classification task

Question

I am working on a classification problem consisting of data with binary features. I am trying to find which features, when equal to 1 give a false negative for a particular class.

To better illustrate my point consider the data below consisting of samples with 5 features along with their GT and predicted classes.

Features |GT Pred
0 0 0 1 0|A  A
0 0 0 0 1|A  A
0 0 0 1 1|A  A
1 0 0 1 1|A  A
1 0 0 0 0|B  B
0 1 0 0 0|B  B
1 1 0 0 0|B  B
0 0 1 0 0|B  B
0 1 1 0 0|C  C
1 0 1 0 0|C  C
1 1 1 0 1|C  A
1 0 1 0 1|C  A

What statistical tests can I perform on the samples with GT label C that would tell me that feature 5 equal to 1 results in a misclassification?

Answer 1

You could try modelling misclassification as a binary variable, and then you have $p$ (number of features) independence tests for two binary variables. You could use the likelihood ratio test or Pearson's $\chi^2$ test (see Wasserman, All of Statistics, Chapter 15).

Note that you will have to correct for the fact that you are doing multiple testing. The most crude approach is Bonferroni's correction, which in case of small $p$ and clear enough dependencies might be enough. Because it is so crude (conservative) you might want to look into a test to correct for false negative rate (Wasserman, AoS, Chapter 10.7)

Update after OP's comment:

One idea to to estimate which single features caused a misclassification could be to estimate the average treatment effect of each one of them, considering each feature as a "treatment" and misclassification as the outcome. For each sample $X, Y$ the outcome $Y$ can be either correctly classified or not and you are interested in whether setting feature $X_{j} = 1$ had an effect. In order to approximate the ATE you need to randomize the treatment, i.e.randomize $X_{j}$ across all samples and then you can estimate

\[ \widehat{\operatorname{ATE}} = \hat{\mathbb{E}} [Y|X_{j} = 1] - \hat{\mathbb{E}} [Y|X_{j} = 0] . \]

The reason that you can use correlations is that, because you randomized the treatment, it is now independent of the potential outcome for each sample.

This method is however quite naive and assumes that single features can flip the classification. Maybe they are related in a more complex way and $X_3=1$ and $X_12=1$ lead to a misclassification but neither of them does separately.