What is the appropriate statistical significance test for multi-class classification?

Question

I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have the results of two classifiers on the same train/test-set (paired observations).

Some sources suggest to use McNemar’s test for binary classification task. However, is there any generalization of McNemar’s test for multi-class classification problem? If so, what would be the appropriate procedure to carry out these tests?

Answer 1

Generalisation of Mcnemars is called Cochran–Mantel–Haenszel test.

There is an implementation in R, but I suppose porting to Python should not be too hard. You can find the R version here.