I am working on a binary classification where I have 4712 records with Label 1 being 1554 records and Label 0 being 3558 records.
When I tried multiple models based on 6,7 and 8 features, I see the below results. Based on the newly added 7th or (7th & 8th) feature, I see an AUC improvement only in one of the models (
LR scikit and
I also come across articles online that says AUC or F1-score aren't strict scoring rules. We could use
log-loss metric but it's only applicable for
logistic regression. but we can't use
log-loss metric for
SVM right? So, is there any common metric which I can use to compare. Am I missing something here?
Does this mean that new feature is helping us improve the performance? But it decreases the performance in other models?
Please note that I split the data into
train and test and did
10 fold CV on train data.
So, how do I know that this newly added 7th feature is really helping in improving the model performance?
update based on answer
from statsmodels.stats.contingency_tables import mcnemar # define contingency table table = [[808,138], # here I added confusion matrix of two models together (I mean based on TP in model 1 is added with TP in model 2 etc) [52, 416]] # calculate mcnemar test result = mcnemar(table, exact=True) # summarize the finding print('statistic=%.3f, p-value=%.3f' % (result.statistic, result.pvalue)) # interpret the p-value alpha = 0.05 if result.pvalue > alpha: print('Same proportions of errors (fail to reject H0)') else: print('Different proportions of errors (reject H0)')