I am working on binary classification problem with 5K records and 60 features.
Through feature selection, I narrowed it down to 14 features.
In existing literature, I see that there are well-known 5 features.
I started my project with an aim to find new feature that can help improve the predictive power of the model
However, I see that with well known features (reported in literature), it produces an AUC of 84-85 and having all my 14 features decreases it to 82-83.
So I tried manual add and drop and found out that if I add only one feature (let's say
magic feature), it increases the
AUC to 85-86.
I see that there is a difference of 1 point in AUC.
1) Is it even useful to be happy that this adds some info to the model?
2) Or me looking at
AUC is not the right way to measure model performance?
3) Does it mean the other new features (9 out of 14) that I selected based on different feature selection/ genetic algorithm aren't that useful? Because my genetic algorithm returned 14 features, so I was assuming that was the best subset but still through my previous experiments I know that model had better performance when it had 5 features. Any suggestions here? What can I do?
4) I am currently using
test split as my training and testing data. I applied
10 fold cv to my data. Should I be doing anything different here?
5) If I add around 16-17 features, I see the
AUC is increased to 87 but this can't be over fitting right? Because if it's overfitting, shouldn't I be seeing the
? I know we haveoccam razor's principle
to keep the model parsimonius but in this case, just having 16-17 features in model is not too complex or heavy. Am I right? Because it's increasing theAUC`. Any suggestions on this?