# Feature Importance - How to choose the number of best features?

What is the standard or what method do you use to select part of features.

For ex: Using random forest, I got the following feature importances:

a : 25.4884726
b : 17.2736393
c : 12.3493490
d : 8.9383737
e : 8.1083837
f : 6.8272717
g : 4.1999999
...
...
...


For final prediction, you need to select only few features from the above. How do you decide the following:

1. No. of features that you choose?
2. Does the score matter when you choose?
• Feature importance is selected by the entropy values in Decision trees. Not sure for RFs. Entropy averaging? – Dawny33 Jul 12 '16 at 6:31
• Go from the top and do cross-validation to observe the score. In doubt, select fewer features. – Gerenuk Jul 13 '16 at 6:36

This is an important problem, since many feature selection methods return feature scores/importances rather than a finite feature set. I currently know three approaches:

• choose the k best attributes (fixed number defined)
• choose the best k * 100% of attributes (relative number of features defined)
• make a cutoff at the biggest difference in feature scores: all features are ordered according to their score and a split is made at the largest difference between one score and the next lower (biggest loss in importance)

You find a nice implementation doing this in R with the FSelector package.

I think, there are some research works tackling exactly this problem and may suggest better approaches, but I hadn't had the time to go deeper into this.

1. No. of features that you choose?

Its depends on the number of classes you have to predict, let's explain this with example: consider we have 5 classes(labels) in our dataset and we choose only one feature, so tree has only one parent node and two leaf node which will accommodate only two classes and hence the accuracy decrease sharply, similarly if we choose only two features then we can only accommodate three classes and so one, so you have to consider more number of features as of classes and there after you have to test it with accuracy till there is no change in accuracy by adding extra features in the model.

2. Does the score(Variable Importance) matter when you choose?

Yes, the score matter when deciding the features that you choose, since its depends on the Variable Importance of a feature is computed as the average decrease in model accuracy on the out of bag samples when the values of the respective feature are randomly permuted, so if you choose only the lower score variables for features then the accuracy sharply decreases.

• Conter-example to your first part : I can predict 4 classes with one feature, let's say the age : baby : [0, 3[, child : [3, 13[, teenager : [13, 18[, adults = [18 and above[. This is because the tree might definitely have more than one parent node and two leaves. – Igor OA Jul 11 '16 at 8:06
• @IgorOA Thanks for pointing it out, you are totally right, I missed it. I have tried creating a decision tree based on your comment but unable to accommodate same features on multiple node in tree, have a look at my github repo R code and fancy decision tree are attached with random sample dataset. – krishna Prasad Jul 12 '16 at 3:36