7
$\begingroup$

GBMs, like random forests, build each tree on a different sample of the dataset and hence, going by the spirit of ensemble models, produce higher accuracies. However, I have not seen GBM being used with dimension sampling at every split of the tree like is common practice with random forests.

Are there some tests that show that dimensional sampling with GBM would decrease its accuracy because of which this is avoided, either in literature form or in practical experience?

$\endgroup$
2
  • 2
    $\begingroup$ What GBM implementations have you looked at? sklearn's GradientBoostingClassifier/Regressor have a max_features parameter and XGBoost has colsample_bylevel and colsample_bytree parameters that control how many features are sampled for each tree / split. $\endgroup$
    – stmax
    Mar 11 '16 at 23:20
  • $\begingroup$ @stmax would you mind posting this comment as an answer? $\endgroup$
    – Nitesh
    Mar 29 '16 at 16:17
2
$\begingroup$

sklearn's GradientBoostingClassifier / GradientBoostingRegressor have a max_features parameter and XGBoost has colsample_bylevel and colsample_bytree parameters that control how many features are sampled for each tree / split.

$\endgroup$
0
$\begingroup$

I have never read (or tried) subsampling techniques on each node of the tree. I do not say that they might not exist, but it looks odd the statement "like is a common practice with random forests".

Other than that, subsampling or bootstrapping the sample for each tree from random forests gives stochastic gradient boosting Friedman, 1999. They give better results with subsampling than without and more details are given also in Elements of Statistical Learning, page 358, 10.12.2 Subsampling.

$\endgroup$
1
  • $\begingroup$ ESL - the Bible. $\endgroup$ Feb 7 '19 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.