how can i compare two classification algorithms?

Question

all,

i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. clearly they are very close! how can i assess if they are statistically different?

i have used mcnemars test:

from mlxtend.evaluate import mcnemar_table from mlxtend.evaluate import mcnemar

lgbm_pred = second_best.predict(x_test)
xg_pred = chosen_model.predict(x_test)

tb = mcnemar_table(y_target=y_test, 
                   y_model1=lgbm_pred, 
                   y_model2=xg_pred)

chi2, p = mcnemar(ary=tb, corrected=True)
print('chi-squared:', chi2)
print('p-value:', p)

output is: chi-squared: 2.25 p-value: 0.13361440253771584, so i would not reject null that models peformance are equal. (hopefully i am using this correctly so pls let me know if i am not.)

i have seen some threads on using 'permuation tests' etc.. but i am unsure how to interpret these and also i thought these tests were only if you have small sample size which i don't.

https://stackoverflow.com/questions/52373318/how-to-compare-roc-auc-scores-of-different-binary-classifiers-and-assess-statist

basically how can i assess which classifier being better? when in an ideal world i would want a classifer which is able to predict has cancer. can i compare precisions of model to model? what is best approach - does anyone also know how i can use permutation tests?

Answer 1

Performance of a model can be judged along several dimensions:

1. Accuracy - Is the train-test validation performing well?

2. Overfitting - Is the difference between training score and the validation score minimal?

3. Efficiency - Is the model light-weight, does it compute and calculate fast?

4. Complexity - Is the model easy to explain, does it use minimal transformations?

5. Deployability - Will I be able to deploy and use the model easily?

Rather than from a pure statistics perspective (which is just dimension 1 & 2) judge the models from all perspecties. If they still perform equally well than they are in fact equal (not surprising for two similiar models performed on the same data set).

Answer 2

Yeah, permutation tests are very slow when having relatively big data.

For this reason, the question you've cited answers that boostrapping is an alternative to the permutation test. If you want to do a permutation test, there's code for it in that question.

However, some practitioners think that doing cross-validation is a better way to compare models. This paper discusses frequentist and bayesian approaches to model comparison via cross-validation.

The main idea is, I compute the cross-validation AUC for my model for 10 different folds. Then I have the AUC of 10 samples for model 1 and the AUC of 10 samples for model 2. Then I perform a test to compare these AUC scores (it can be a permutation test or a t-test).