Does using unimportant features hurt accuracy?

Question

I'm using the scikit-learn gradient boosting classifier found here.

If I run the classifier on the same data without seeding the random number generator, I get different feature importances, and notably, different ones are 0 each time. If I just look at the features where the feature importance is < .005, most of the same ones appear, but there is a little variation there too.

My question is, does this have an impact other than computation time? If it were to hurt accuracy, I imagine that would be due to over-fitting. However, if the classifier already identifies the relatively unimportant features, then doesn't that take care of the problem?

If it does hurt accuracy, how should I select which features to get rid of?

Answer 1

I used the gradient boosting classifier in a project a while ago. We had about 130 features and since the system had to be implemented on a microcontroller I tried to reduce the number of features as far as possible.

Here is a plot showing the performance (F1-score) of GBC models trained with the top n most important features:

From that image it looks like "good-enough performance" is already reached with only 30 out of 130 features, while the other 100 (unimportant) features don't seem to have much positive nor negative influence.

Only when you zoom in a bit on the Y-axis you see this:

This shows that the performance reaches its peak (0.93) at 90 out of 130 features. So 40 features are unimportant if you want the best model.

To answer your question "does this have an impact other than computation time?": it does have an impact on performance, since the performance decreases again with more than 90 features, but only slightly.

To answer your second question "If it does hurt accuracy, how should I select which features to get rid of?":

I did the following:

1. Select the top n features
2. Use a cross-validated grid search to train a GBC model on the top n features
3. Repeat with the top n+1,... features
4. Select the model / features with the best score

How to select the top n features in step 1):

I actually didn't use GBC for feature selection, but random forests. I don't know why, but my experiments have shown that selecting features with RF and then training a GBC model on them works better than selecting features with GBC and then training a GBC model on them.

Answer 2

Random Forests and GBT are robust against redundant features, so accuracy is unlikely to be negatively affected. That some features have very low importance is evidence of them being ignored. Because the trees are constructed in a greedy fashion, with randomized starting points different features might end up being ignored. If feature Foo and Bar contain redundant information, then Bar will be ignored if Foo was chosen first - and vica versa.