1
$\begingroup$

I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have the results of two classifiers on the same train/test-set (paired observations).

Some sources suggest to use McNemar’s test for binary classification task. However, is there any generalization of McNemar’s test for multi-class classification problem? If so, what would be the appropriate procedure to carry out these tests?

$\endgroup$
2
$\begingroup$

Generalisation of Mcnemars is called Cochran–Mantel–Haenszel test.

There is an implementation in R, but I suppose porting to Python should not be too hard. You can find the R version here.

$\endgroup$
1
  • $\begingroup$ Hello in which section to find the code in R? Thanks $\endgroup$
    – abdoulsn
    Aug 4 at 23:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.