7
$\begingroup$

I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do it? Will McNemar's test be enough to validate it? Which statistical test will tell me which classifier is better than the rest? And how?

$\endgroup$
2
$\begingroup$

Cochran's Q test

Is a generalisation of the McNemars test and can be used to see if there is a truly better classifier for the metric chosen. You can ofcourse also do pairwise Mcnemare test and draw conclusions from there.

NOTE:

These things are expensive

| improve this answer | |
$\endgroup$
  • $\begingroup$ remember that the Cochran's Q test assumes dichotomous dependent variable. So for multi-class dependent variable >2 classes won't be supported. $\endgroup$ – Prometheus Dec 31 '19 at 3:06
1
$\begingroup$

Janez Demsar has published an article concerning comparison of different classifiers. When you're using multiple datasets to check which algorithm performs best, assuming, that quality measurements come from normal distribution can be risky, so ANOVA is not necessarily recommended. (With ~8k citations it's a canonical article about comparison of classifiers.)

Non-parametric tests (such as Friedmann test) can be used to obtain F-score. Then post-hoc can be applied to test whether one algorithm outperforms others. Bear in mind, that checking n different algorithms implies different tests than checking an algorithm against reference (well known algorithm). In the former case Nemenyi test performs well, in the latter Bonferroni-Dunn test (chapter 3.2.2).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.