How to distinguish informative and non-informative feature - Feature importance?

Question

I have a dataset with 5K records focused on binary classification problem. I have more than 60 features in my dataset. When I used Xgboost, I got the below Feature Importance plot. However I am not sure how to find out whether all of these are informative?

Questions

1) Yes, I can select top 15/20/25 etc. But is that how it is done? Is there any minimum F-score that we ought to look for?

2) Or is it like I select top 10 features, check for accuracy and again add 2-3 features in every round and verify accuracy manually. Is this how it is done?

3) How would you people go about it? I tried with full dataset, the accuracy is only about 86%. when I tried with 15-20 features, it's only about 84. So manual feature selection is the only way to improve further?

Can you help me please?

Answer 1

My approach is quite different. After the creation of feature importance, I usually formulate the question "Is a feature of low importance really non-informative or its informative nature is shadowed by other features?"

How do I answer this question?

I use a simple method - decreasing the number of features in each iteration* (a greater chance that the most important features are not between them), adding a dummy feature containing random numbers, and creating a new feature importance. The random dummy feature is non-informative by definition, so assumption, that features of importance less than this dummy feature are non-informative is IMHO not so strange. So, the next step is to remove these features.

Then I usually make a selection starting from the other side. Sometimes more valuable, more important features, being surely informative, cause overfitting and removing them increases accuracy.

*) The default partition of features in each iteration (colsample_bytree) is equal to 1, but it may be set in the stage of feature selection to a much lower value of the range 0.2-0.4.