2
$\begingroup$

What I don't understand about gradient boost is, doesn't lowering height of the tree means we use fewer features in our model?

From my model I get the highest accuracy when the depth is one. Meaning there is just root node at my trees, and uses one feature. How can a model that uses one feature gives such accuracy?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

I find it really hard to imagine how tree-based boosting works. I think there are two important components:

  1. Boosting can be seen as an ensemble method, so in essence it is not one "small" tree, but many (in fact a huge number of) small trees which learn together.
  2. During learning, there is feedback on the learning process ("adaptive boosting"). So observations which are "hard to predict", receive a higher weight (more attantion if you like). This allows the ensemble to learn "deeper" than other methods.

The xgboost docs come with a nice introduction on boosted trees.

Chapter 10 of "Elements of Statistical Learning" covers boosting. It is worth a look. Below is the AdaBoost routine as covered by the book.

enter image description here

There are also other boosting methods which do not work with trees. Here is a simple L2 boosting routine implemented in R.

Improve this answer
$\endgroup$
5
  • $\begingroup$ First you have a tree, and you generate better versions of that tree. That's ok until here. What I don't understand is, if you have a depth 1 it means you will use only one feature. And you generate your solution according to that tree. How can one feature give us a result? $\endgroup$
    – J.Smith
    Commented Aug 18, 2019 at 18:04
  • $\begingroup$ This is the ensemble part as I understand it, but to be honest, I can‘t give you a good answer on that. I still struggle to get the details... $\endgroup$
    – Peter
    Commented Aug 18, 2019 at 18:08
  • $\begingroup$ @J.Smith, "...and you generate better versions of that tree" is incorrect: you generate new trees, in addition to the old ones, and the final prediction (in gradient boosting) is the sum of all the trees' contributions. $\endgroup$
    – Ben Reiniger ♦
    Commented Aug 18, 2019 at 23:38
  • $\begingroup$ @BenReiniger Hello, thank you for correction. What about feature selection? As I know unlike random forests there is no random feature selection. So do we have the same feature for every tree? $\endgroup$
    – J.Smith
    Commented Aug 19, 2019 at 1:24
  • 1
    $\begingroup$ @J.Smith, both random forest and gradient boosting can subset the features to examine (for each tree or each node). $\endgroup$
    – Ben Reiniger ♦
    Commented Aug 20, 2019 at 20:45
0
$\begingroup$

In boosting, a low biased & low variance model is ensembled by additively combining high biased & low variant models. Here, biases are gradually diminished while keeping variances low. Therefore, a tree with a single depth, that is highly biased, isn't an issue because additively errors will be reduced.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.