Full Disclosure:
I did a semi-cross post of this question due to low traffic on Cross Validated. Once I get an answer on any of the two questions, I will link the answer back to the respective other.
tl;dr
For multiclass classifiers, can you apply McNemar's Test to determine, whether two classifiers are significantly different in how they categorize the same data? Or is McNemar limited to 2-class problems?
Detailed Question
I need to determine whether a number of classifiers are pairwise significantly different in their predictions. I found several sources mentioning McNemar as suited for this. Example sources:
However, I am not sure if these sources assumed binary classifiers.
Now I would like to know if I can apply McNemar's test my multiclass case. To illustrate let me give you an example. For that let's generate a bit of random data
>>> # number of categories
... k = 4
>>>
>>> # random data representing the ground truth in k categories
... ground_truth = np.random.randint(0,k,1000)
>>> # random data representing predictions by two different classifiers
... preds1 = np.random.randint(0,k,1000)
>>> preds2 = np.random.randint(0,k,1000)
Now, given this data, can I apply McNemar?
>>> # binary arrays coding for whether a prediction did match with the ground truth
... results1 = preds1 == ground_truth
>>> results2 = preds2 == ground_truth
>>>
>>> table = np.bincount(2 * (results1) + (results2), minlength=2*2).reshape(2, 2)
>>>
>>> print(table)
[[559 186]
[186 69]]
>>> from statsmodels.stats.contingency_tables import mcnemar
>>> print(mcnemar(table))
pvalue 1.0
statistic 186.0
Would the test be applied correctly like this? Or is McNemar limited to 2-class classifiers only?
My classes are imbalanced btw. if that is relevant.
