Decision Trees Should We Discard Low Importance Features?

Question

I just started to work with feature selection. Let's say I have a decision tree model. I get its feature importances by tree.feature_importances_.

In my model out of around 30 features, 20 of them has importance value of zero. Does that mean that I should discard those low importance value features from my model? As I understood the answer is no, but I don't know the reason behind it. Can anyone explain?

Answer 1

As for many questions, the answer is "it depends":

• features which have a low individual importance may still add predictive power to your model, because the model benefits from combining their information together with information of other features.
• However they may introduce noise in the model and cause overfitting, thus decreasing the performance of the model.

The best way to answer your question is to experiment:

1. order the features by decreasing importance
2. loop from say 5 features to 30, each time selecting the top N features by importance, and training/testing a model based on this subset of features.
3. plot the performance

You're likely to observe that the performance increases quite a lot at the start for each "important" feature added, then slows down as feature importance decreases and probably doesn't increase at all at some point, possibly even decreasing a bit.

Answer 2

Introduction to Data Science with Python book says

However, if a feature has a low feature_importance , it doesn’t mean that this feature is uninformative. It only means that the feature was not picked by the tree, likely because another feature encodes the same information.

So in that specific model, you should be able to discard because there is an other variable encodes that information. But another model may use it.