# Gradient boosting how can accuracy increase when we lower the depth of tree?

What I don't understand about gradient boost is, doesn't lowering height of the tree means we use fewer features in our model?

From my model I get the highest accuracy when the depth is one. Meaning there is just root node at my trees, and uses one feature. How can a model that uses one feature gives such accuracy?

I find it really hard to imagine how tree-based boosting works. I think there are two important components:

1. Boosting can be seen as an ensemble method, so in essence it is not one "small" tree, but many (in fact a huge number of) small trees which learn together.
2. During learning, there is feedback on the learning process ("adaptive boosting"). So observations which are "hard to predict", receive a higher weight (more attantion if you like). This allows the ensemble to learn "deeper" than other methods.

The xgboost docs come with a nice introduction on boosted trees.

Chapter 10 of "Elements of Statistical Learning" covers boosting. It is worth a look. Below is the AdaBoost routine as covered by the book.

There are also other boosting methods which do not work with trees. Here is a simple L2 boosting routine implemented in R.

• First you have a tree, and you generate better versions of that tree. That's ok until here. What I don't understand is, if you have a depth 1 it means you will use only one feature. And you generate your solution according to that tree. How can one feature give us a result? – J.Smith Aug 18 '19 at 18:04
• This is the ensemble part as I understand it, but to be honest, I can‘t give you a good answer on that. I still struggle to get the details... – Peter Aug 18 '19 at 18:08
• @J.Smith, "...and you generate better versions of that tree" is incorrect: you generate new trees, in addition to the old ones, and the final prediction (in gradient boosting) is the sum of all the trees' contributions. – Ben Reiniger Aug 18 '19 at 23:38
• @BenReiniger Hello, thank you for correction. What about feature selection? As I know unlike random forests there is no random feature selection. So do we have the same feature for every tree? – J.Smith Aug 19 '19 at 1:24
• @J.Smith, both random forest and gradient boosting can subset the features to examine (for each tree or each node). – Ben Reiniger Aug 20 '19 at 20:45

In boosting, a low biased & low variance model is ensembled by additively combining high biased & low variant models. Here, biases are gradually diminished while keeping variances low. Therefore, a tree with a single depth, that is highly biased, isn't an issue because additively errors will be reduced.