How to present a statistical justification for the choice of models with approximate accuracies?

Question

In an experiment involving the comparison of classification algorithms, how can I assess whether there are statistically significant differences between the analyzed models? For example, the following models were tested, and presented the respective accuracies: CatBoost 79.32%; XGBoost 77.05%; SVM 76.77%; LightGBM 76.48%; RF 73.93%; KNN 70.25%. What method can I use to validate the choice of CatBoost over XGBoost, for example? Or let them know that KNN can be just as useful as an SVM.

Details: equal dataset for all models; training/test (75/25); default values for all model; multiclass classification.

Is there any way to justify the choice for one of the methods above? Something that escapes the trivial "choose the one with the highest accuracy".

Is it possible to employ some hypothesis test? (e.g. Chi-square; Wilcoxon; ANOVA)

Answer 1

A widely accepted approach for your problem is using the Critical Difference Diagram. At this time, the original paper has 12274 citations (and counting).

Implementations:

• Python;
• Julia;
• R.