Full Disclosure:

I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other.


For multiclass classifiers, can you apply McNemar's Test to determine, whether two classifiers are significantly different in how they categorize the same data? Or is McNemar limited to 2-class problems?

Detailed Question

I need to determine whether a number of classifiers are pairwise significantly different in their predictions. I found several sources mentioning McNemar as suited for this. Example sources:

example 1

example 2

However, I am not sure if these sources assumed binary classifiers.

Now I would like to know if I can apply McNemar's test my multiclass case. To illustrate let me give you an example. For that let's generate a bit of random data

>>> # number of categories
... k = 4
>>> # random data representing the ground truth in k categories
... ground_truth = np.random.randint(0,k,1000)
>>> # random data representing predictions by two different classifiers
... preds1 = np.random.randint(0,k,1000)
>>> preds2 = np.random.randint(0,k,1000)

Now, given this data, can I apply McNemar?

>>> # binary arrays coding for whether a prediction did match with the ground truth
... results1 = preds1 == ground_truth
>>> results2 = preds2 == ground_truth
>>> table = np.bincount(2 * (results1) + (results2), minlength=2*2).reshape(2, 2)
>>> print(table)
[[559 186]
 [186  69]]
>>> from statsmodels.stats.contingency_tables import mcnemar
>>> print(mcnemar(table))
pvalue      1.0
statistic   186.0

Would the test be applied correctly like this? Or is McNemar limited to 2-class classifiers only?

My classes are imbalanced btw. if that is relevant.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.