i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. clearly they are very close! how can i assess if they are statistically different?
i have used mcnemars test:
from mlxtend.evaluate import mcnemar_table from mlxtend.evaluate import mcnemar
lgbm_pred = second_best.predict(x_test) xg_pred = chosen_model.predict(x_test) tb = mcnemar_table(y_target=y_test, y_model1=lgbm_pred, y_model2=xg_pred) chi2, p = mcnemar(ary=tb, corrected=True) print('chi-squared:', chi2) print('p-value:', p)
output is: chi-squared: 2.25 p-value: 0.13361440253771584, so i would not reject null that models peformance are equal. (hopefully i am using this correctly so pls let me know if i am not.)
i have seen some threads on using 'permuation tests' etc.. but i am unsure how to interpret these and also i thought these tests were only if you have small sample size which i don't.
basically how can i assess which classifier being better? when in an ideal world i would want a classifer which is able to predict has cancer. can i compare precisions of model to model? what is best approach - does anyone also know how i can use permutation tests?