1
$\begingroup$

Full Disclosure:

I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other.


tl;dr

For multiclass classifiers, can you apply McNemar's Test to determine, whether two classifiers are significantly different in how they categorize the same data? Or is McNemar limited to 2-class problems?


Detailed Question

I need to determine whether a number of classifiers are pairwise significantly different in their predictions. I found several sources mentioning McNemar as suited for this. Example sources:

example 1

example 2

However, I am not sure if these sources assumed binary classifiers.

Now I would like to know if I can apply McNemar's test my multiclass case. To illustrate let me give you an example. For that let's generate a bit of random data

>>> # number of categories
... k = 4
>>> 
>>> # random data representing the ground truth in k categories
... ground_truth = np.random.randint(0,k,1000)
>>> # random data representing predictions by two different classifiers
... preds1 = np.random.randint(0,k,1000)
>>> preds2 = np.random.randint(0,k,1000)

Now, given this data, can I apply McNemar?

>>> # binary arrays coding for whether a prediction did match with the ground truth
... results1 = preds1 == ground_truth
>>> results2 = preds2 == ground_truth
>>> 
>>> table = np.bincount(2 * (results1) + (results2), minlength=2*2).reshape(2, 2)
>>> 
>>> print(table)
[[559 186]
 [186  69]]
>>> from statsmodels.stats.contingency_tables import mcnemar
>>> print(mcnemar(table))
pvalue      1.0
statistic   186.0

Would the test be applied correctly like this? Or is McNemar limited to 2-class classifiers only?

My classes are imbalanced btw. if that is relevant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.