I used the gradient boosting classifier in a project a while ago. We had about 130 features and since the system had to be implemented on a microcontroller I tried to reduce the number of features as far as possible.
Here is a plot showing the performance (F1-score) of GBC models trained with the top n most important features:
From that image it looks like "good-enough performance" is already reached with only 30 out of 130 features, while the other 100 (unimportant) features don't seem to have much positive nor negative influence.
Only when you zoom in a bit on the Y-axis you see this:
This shows that the performance reaches its peak (0.93) at 90 out of 130 features. So 40 features are unimportant if you want the best model.
To answer your question "does this have an impact other than computation time?": it does have an impact on performance, since the performance decreases again with more than 90 features, but only slightly.
To answer your second question "If it does hurt accuracy, how should I select which features to get rid of?":
I did the following:
- Select the top n features
- Use a cross-validated grid search to train a GBC model on the top n features
- Repeat with the top n+1,... features
- Select the model / features with the best score
How to select the top n features in step 1):
I actually didn't use GBC for feature selection, but random forests. I don't know why, but my experiments have shown that selecting features with RF and then training a GBC model on them works better than selecting features with GBC and then training a GBC model on them.