I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records.
When I tried multiple models based on 6,7 and 8 features, I see the below results. Based on the newly added 7th or (7th & 8th) feature, I see an AUC improvement only in one of the models (LR scikit and Xgboost).
I also come across articles online that says AUC or F1-score aren't strict scoring rules. We could use log-loss metric but it's only applicable for logistic regression. but we can't use log-loss metric for Xgboost or RF or SVM right? So, is there any common metric which I can use to compare. Am I missing something here?
Does this mean that new feature is helping us improve the performance? But it decreases the performance in other models?
Please note that I split the data into train and test and did 10 fold CV on train data.
So, how do I know that this newly added 7th feature is really helping in improving the model performance?
update based on answer
from statsmodels.stats.contingency_tables import mcnemar
# define contingency table
table = [[808,138], # here I added confusion matrix of two models together (I mean based on TP in model 1 is added with TP in model 2 etc)
[52, 416]]
# calculate mcnemar test
result = mcnemar(table, exact=True)
# summarize the finding
print('statistic=%.3f, p-value=%.3f' % (result.statistic, result.pvalue))
# interpret the p-value
alpha = 0.05
if result.pvalue > alpha:
print('Same proportions of errors (fail to reject H0)')
else:
print('Different proportions of errors (reject H0)')
