I have a dataset with 5K records focused on binary classification problem. I have more than 60 features in my dataset. When I used Xgboost, I got the below Feature Importance plot. However I am not sure how to find out whether all of these are informative?

enter image description here


1) Yes, I can select top 15/20/25 etc. But is that how it is done? Is there any minimum F-score that we ought to look for?

2) Or is it like I select top 10 features, check for accuracy and again add 2-3 features in every round and verify accuracy manually. Is this how it is done?

3) How would you people go about it? I tried with full dataset, the accuracy is only about 86%. when I tried with 15-20 features, it's only about 84. So manual feature selection is the only way to improve further?

Can you help me please?


1 Answer 1


My approach is quite different. After the creation of feature importance, I usually formulate the question "Is a feature of low importance really non-informative or its informative nature is shadowed by other features?"

How do I answer this question?

I use a simple method - decreasing the number of features in each iteration* (a greater chance that the most important features are not between them), adding a dummy feature containing random numbers, and creating a new feature importance. The random dummy feature is non-informative by definition, so assumption, that features of importance less than this dummy feature are non-informative is IMHO not so strange. So, the next step is to remove these features.

Then I usually make a selection starting from the other side. Sometimes more valuable, more important features, being surely informative, cause overfitting and removing them increases accuracy.

*) The default partition of features in each iteration (colsample_bytree) is equal to 1, but it may be set in the stage of feature selection to a much lower value of the range 0.2-0.4.

  • $\begingroup$ Apppreciate yoyur time and inputs. Upvoted. Will try and keey you posted $\endgroup$
    – The Great
    Dec 15, 2019 at 23:14
  • $\begingroup$ Is there anyway to get a p-value for my ranked features? $\endgroup$
    – The Great
    Dec 18, 2019 at 23:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.