# Gradient boosting how can accuracy increase when we lower the depth of tree?

What I don't understand about gradient boost is, doesn't lowering height of the tree means we use fewer features in our model?

From my model I get the highest accuracy when the depth is one. Meaning there is just root node at my trees, and uses one feature. How can a model that uses one feature gives such accuracy?

I find it really hard to imagine how tree-based boosting works. I think there are two important components:

1. Boosting can be seen as an ensemble method, so in essence it is not one "small" tree, but many (in fact a huge number of) small trees which learn together.
2. During learning, there is feedback on the learning process ("adaptive boosting"). So observations which are "hard to predict", receive a higher weight (more attantion if you like). This allows the ensemble to learn "deeper" than other methods.

The xgboost docs come with a nice introduction on boosted trees.

Chapter 10 of "Elements of Statistical Learning" covers boosting. It is worth a look. Below is the AdaBoost routine as covered by the book.

There are also other boosting methods which do not work with trees. Here is a simple L2 boosting routine implemented in R.

• First you have a tree, and you generate better versions of that tree. That's ok until here. What I don't understand is, if you have a depth 1 it means you will use only one feature. And you generate your solution according to that tree. How can one feature give us a result? Commented Aug 18, 2019 at 18:04
• This is the ensemble part as I understand it, but to be honest, I can‘t give you a good answer on that. I still struggle to get the details... Commented Aug 18, 2019 at 18:08
• @J.Smith, "...and you generate better versions of that tree" is incorrect: you generate new trees, in addition to the old ones, and the final prediction (in gradient boosting) is the sum of all the trees' contributions. Commented Aug 18, 2019 at 23:38
• @BenReiniger Hello, thank you for correction. What about feature selection? As I know unlike random forests there is no random feature selection. So do we have the same feature for every tree? Commented Aug 19, 2019 at 1:24
• @J.Smith, both random forest and gradient boosting can subset the features to examine (for each tree or each node). Commented Aug 20, 2019 at 20:45

In boosting, a low biased & low variance model is ensembled by additively combining high biased & low variant models. Here, biases are gradually diminished while keeping variances low. Therefore, a tree with a single depth, that is highly biased, isn't an issue because additively errors will be reduced.