2
$\begingroup$

In an experiment involving the comparison of classification algorithms, how can I assess whether there are statistically significant differences between the analyzed models? For example, the following models were tested, and presented the respective accuracies: CatBoost 79.32%; XGBoost 77.05%; SVM 76.77%; LightGBM 76.48%; RF 73.93%; KNN 70.25%. What method can I use to validate the choice of CatBoost over XGBoost, for example? Or let them know that KNN can be just as useful as an SVM.

Details: equal dataset for all models; training/test (75/25); default values for all model; multiclass classification.

Is there any way to justify the choice for one of the methods above? Something that escapes the trivial "choose the one with the highest accuracy".

Is it possible to employ some hypothesis test? (e.g. Chi-square; Wilcoxon; ANOVA)

$\endgroup$

1 Answer 1

1
$\begingroup$

A widely accepted approach for your problem is using the Critical Difference Diagram. At this time, the original paper has 12274 citations (and counting).

Implementations:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.