1
$\begingroup$

I am working on binary classification problem with 5K records and 60 features.

Through feature selection, I narrowed it down to 14 features.

In existing literature, I see that there are well-known 5 features.

I started my project with an aim to find new feature that can help improve the predictive power of the model

However, I see that with well known features (reported in literature), it produces an AUC of 84-85 and having all my 14 features decreases it to 82-83.

So I tried manual add and drop and found out that if I add only one feature (let's say magic feature), it increases the AUC to 85-86.

I see that there is a difference of 1 point in AUC.

1) Is it even useful to be happy that this adds some info to the model?

2) Or me looking at AUC is not the right way to measure model performance?

3) Does it mean the other new features (9 out of 14) that I selected based on different feature selection/ genetic algorithm aren't that useful? Because my genetic algorithm returned 14 features, so I was assuming that was the best subset but still through my previous experiments I know that model had better performance when it had 5 features. Any suggestions here? What can I do?

4) I am currently using train and test split as my training and testing data. I applied 10 fold cv to my data. Should I be doing anything different here?

5) If I add around 16-17 features, I see the AUC is increased to 87 but this can't be over fitting right? Because if it's overfitting, shouldn't I be seeing the AUC as97-100or just100? I know we haveoccam razor's principleto keep the model parsimonius but in this case, just having 16-17 features in model is not too complex or heavy. Am I right? Because it's increasing theAUC`. Any suggestions on this?

$\endgroup$
1
$\begingroup$

A lot of questions here. Here are some thoughts.

  • should you be happy about a 1 point increase in AUC? Yes. An effect can be genuine, but small. A 1 point advantage is still an improvement. But do I trust that outcome? Not sure.
  • you need some more data. Your sample size is not large. Furthermore, cross-validation is a wonderful thing, but you have been running a lot of tests on the same small data set, so cross-validation notwithstanding -- we really don't know how your classifier will perform on brand new, unseen data.

  • point 3. This sounds like an issue with over-fitting.

  • point 4. I'm not sure what you are doing here. Cross-validation is an alternative to "train" and "test" sets, since each fold acts as the "test" set to the model fit on the "Non-fold" part of the data. Did you mean to say that you were doing CV on the train part, and then used the "test" part for a final check on performance? That would be a good thing to do, but if you have been using the same test set to check every model you have tried, then that test set is beginning to look like a training set. You will need another "test" set.

  • point 5. You don't need a ridiculously high AUC to be guilty of over-fitting. Over-fitting happens when you fit features to the noise in your data set. An over-fit model will have a higher mean squared error on a fresh test set than the optimal model, although it will do better on the training set. Having 16 to 18 features is too many if, in fact, the outcome is well explained with only 5.

  • Given that you have reduced your candidate features to 14, you now have a much smaller problem, and it should be possible to examine the features from a subject matter perspective. Which features would a SME recommend you to retain? And you can also examine correlations between the 14 features and check for redundancy that way. With a small data problem (which is what this is), you can work to understand your data directly. That approach might yield some interesting insights.

  • You might want to use a different method of model selection than maximal AUC. This paper discusses the merits of AUC and offers an alternative. Could be interesting.

$\endgroup$
  • $\begingroup$ Hi @Placidia - thanks for your response. upvoted.a quick question regarding your points. 1) For point 1, how can I trust/verify that outcome? Is there anyway to double check/verify that 1 point increase is only due to this new feature 2) for point 4, Yes I split my data into train (70%) and test (30%). later I applied CV on train data only. Then use test data to predict the outcome. I am hearing about bootstrap validation. Do you think that will help? $\endgroup$ – The Great Jan 6 at 13:40
  • $\begingroup$ Is it possible that doing bootstrap validation can help increase model performance or something? Am trying to seek inputs from you and people in the forum and make changes to my model/code so that there will be no more options left to try it out. Is there anything that you would like to suggest? Because it's not possible that certain features don't add predictive power. Based on SME, we feel certain features are really useful but they don't help. Can it be an issue due to data extraction? $\endgroup$ – The Great Jan 6 at 13:52
  • $\begingroup$ For point 1, I think the safest way is to test on a brand new sample, if that is possible. Build the AUC with 6 features, and then with the legacy five. Compare the outcomes. I don't know what model you are using, but if it has a likelihood function, you can compare the likelihoods for your two models. $\endgroup$ – Placidia Jan 6 at 14:11
  • $\begingroup$ Hi, I am using Xgboost model. Though logistic regression, It doesn't perform well. In addition, Have you heard of net reclassification index? Do you know is there any python implementation for it? $\endgroup$ – The Great Jan 6 at 14:28
  • $\begingroup$ Sorry. I can't help you on that. $\endgroup$ – Placidia Jan 6 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.