Boosting methods (such as the popular xgboost) do not tend to overfit when we use many iterations - Schapire and Freund. Are they also resistant to overfitting when we feed them with a large number of features (where some of the features are not very useful?) If so, is there a theoretic connection between these two desirable properties?

(related to https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality)


I can't speak from a theoretical perspective, but I can say that in practice overfitting with a large number of features is not an issue with gradient boosting. I work at a huge financial institution that has been doing credit risk modeling for decades, and for the past decade or so they have almost exclusively been doing their variable selection using gradient boosting. Gradient boosting (along with any tree-based method) can be used to find relative feature importances (based on how much error is reduced after each split).

Statisticians where I work will often use gradient boosting to narrow down 2,000 or so features into a more manageable 10-15 by taking the top ranked features. These features are then fed into a logistic regression model, which is more interpretable than gradient boosting. Even on the hold-out partition, the gradient boosting algorithm built on the 2,000 features generally outperforms the logistic regression model. The gradient boosting model is thus used for variable selection and predictive performance benchmarking.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.