I am working on a classification problem consisting of data with binary features. I am trying to find which features, when equal to 1 give a false negative for a particular class.

To better illustrate my point consider the data below consisting of samples with 5 features along with their GT and predicted classes.

Features |GT Pred
0 0 0 1 0|A  A
0 0 0 0 1|A  A
0 0 0 1 1|A  A
1 0 0 1 1|A  A
1 0 0 0 0|B  B
0 1 0 0 0|B  B
1 1 0 0 0|B  B
0 0 1 0 0|B  B
0 1 1 0 0|C  C
1 0 1 0 0|C  C
1 1 1 0 1|C  A
1 0 1 0 1|C  A

What statistical tests can I perform on the samples with GT label C that would tell me that feature 5 equal to 1 results in a misclassification?


1 Answer 1


You could try modelling misclassification as a binary variable, and then you have $p$ (number of features) independence tests for two binary variables. You could use the likelihood ratio test or Pearson's $\chi^2$ test (see Wasserman, All of Statistics, Chapter 15).

Note that you will have to correct for the fact that you are doing multiple testing. The most crude approach is Bonferroni's correction, which in case of small $p$ and clear enough dependencies might be enough. Because it is so crude (conservative) you might want to look into a test to correct for false negative rate (Wasserman, AoS, Chapter 10.7)

Update after OP's comment:

One idea to to estimate which single features caused a misclassification could be to estimate the average treatment effect of each one of them, considering each feature as a "treatment" and misclassification as the outcome. For each sample $X, Y$ the outcome $Y$ can be either correctly classified or not and you are interested in whether setting feature $X_{j} = 1$ had an effect. In order to approximate the ATE you need to randomize the treatment, i.e.randomize $X_{j}$ across all samples and then you can estimate

\[ \widehat{\operatorname{ATE}} = \hat{\mathbb{E}} [Y|X_{j} = 1] - \hat{\mathbb{E}} [Y|X_{j} = 0] . \]

The reason that you can use correlations is that, because you randomized the treatment, it is now independent of the potential outcome for each sample.

This method is however quite naive and assumes that single features can flip the classification. Maybe they are related in a more complex way and $X_3=1$ and $X_12=1$ lead to a misclassification but neither of them does separately.

  • $\begingroup$ Thanks. The problem with independence tests is that it doesn't tell me which binary features when equal to 1 lead to misclassification -- it tells me which features are important for deciding between correctly classsified and misclassified samples. How would I be able to whittle the results of independence tests to give me features that are more likely to equal 1 in misclassified samples than in correctly classified samples and exclude features that are more likely to equal 1 in correctly classified than misclassified samples? $\endgroup$
    – migs
    Jun 27, 2021 at 6:06
  • $\begingroup$ Sorry, it seems like I answered too quickly and missed the point. I've updated my answer with an idea to use average treatment effects, but it's a bit speculative. $\endgroup$
    – Miguel
    Jun 27, 2021 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.