Threshold to consider new feature as a new finding to a model?

Question

I am working on binary classification problem with 5K records and 60 features.

Through feature selection, I narrowed it down to 14 features.

In existing literature, I see that there are well-known 5 features.

I started my project with an aim to find new feature that can help improve the predictive power of the model

However, I see that with well known features (reported in literature), it produces an AUC of 84-85 and having all my 14 features decreases it to 82-83.

So I tried manual add and drop and found out that if I add only one feature (let's say magic feature), it increases the AUC to 85-86.

I see that there is a difference of 1 point in AUC.

1) Is it even useful to be happy that this adds some info to the model?

2) Or me looking at AUC is not the right way to measure model performance?

3) Does it mean the other new features (9 out of 14) that I selected based on different feature selection/ genetic algorithm aren't that useful? Because my genetic algorithm returned 14 features, so I was assuming that was the best subset but still through my previous experiments I know that model had better performance when it had 5 features. Any suggestions here? What can I do?

4) I am currently using train and test split as my training and testing data. I applied 10 fold cv to my data. Should I be doing anything different here?

5) If I add around 16-17 features, I see the AUC is increased to 87 but this can't be over fitting right? Because if it's overfitting, shouldn't I be seeing the AUC as97-100or just100? I know we haveoccam razor's principleto keep the model parsimonius but in this case, just having 16-17 features in model is not too complex or heavy. Am I right? Because it's increasing theAUC`. Any suggestions on this?

Answer 1

A lot of questions here. Here are some thoughts.

• should you be happy about a 1 point increase in AUC? Yes. An effect can be genuine, but small. A 1 point advantage is still an improvement. But do I trust that outcome? Not sure.

• you need some more data. Your sample size is not large. Furthermore, cross-validation is a wonderful thing, but you have been running a lot of tests on the same small data set, so cross-validation notwithstanding -- we really don't know how your classifier will perform on brand new, unseen data.

• point 3. This sounds like an issue with over-fitting.

• point 4. I'm not sure what you are doing here. Cross-validation is an alternative to "train" and "test" sets, since each fold acts as the "test" set to the model fit on the "Non-fold" part of the data. Did you mean to say that you were doing CV on the train part, and then used the "test" part for a final check on performance? That would be a good thing to do, but if you have been using the same test set to check every model you have tried, then that test set is beginning to look like a training set. You will need another "test" set.

• point 5. You don't need a ridiculously high AUC to be guilty of over-fitting. Over-fitting happens when you fit features to the noise in your data set. An over-fit model will have a higher mean squared error on a fresh test set than the optimal model, although it will do better on the training set. Having 16 to 18 features is too many if, in fact, the outcome is well explained with only 5.

• Given that you have reduced your candidate features to 14, you now have a much smaller problem, and it should be possible to examine the features from a subject matter perspective. Which features would a SME recommend you to retain? And you can also examine correlations between the 14 features and check for redundancy that way. With a small data problem (which is what this is), you can work to understand your data directly. That approach might yield some interesting insights.

• You might want to use a different method of model selection than maximal AUC. This paper discusses the merits of AUC and offers an alternative. Could be interesting.