I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do it? Will McNemar's test be enough to validate it? Which statistical test will tell me which classifier is better than the rest? And how?
Is a generalisation of the McNemars test and can be used to see if there is a truly better classifier for the metric chosen. You can ofcourse also do pairwise Mcnemare test and draw conclusions from there.
These things are expensive
Janez Demsar has published an article concerning comparison of different classifiers. When you're using multiple datasets to check which algorithm performs best, assuming, that quality measurements come from normal distribution can be risky, so ANOVA is not necessarily recommended. (With ~8k citations it's a canonical article about comparison of classifiers.)
Non-parametric tests (such as Friedmann test) can be used to obtain F-score. Then post-hoc can be applied to test whether one algorithm outperforms others. Bear in mind, that checking n different algorithms implies different tests than checking an algorithm against reference (well known algorithm). In the former case Nemenyi test performs well, in the latter Bonferroni-Dunn test (chapter 3.2.2).