# How to interpret gradient descent in boosting ensembles?

I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) sequentially -- each subsequent one is learning from the errors of the previous ones (by upweighting the misclassified examples, if I see correctly) and combining the results.

(Subquestion: does this combination mean that we use all the subsequently trained constituent estimators, maybe with different weights, or we just take the final one, which is assumed to be the most accurate?).

However, I cannot figure out how gradient descent and learning rate comes into the picture here. Trees themselves are not gradient based learners, and combining the output (either way) doesn't require any optimization. So what is its role?

• It seems you have mixed Adaptive Boosting and Gradient Boosting. Please read this or any other reference and update the question accordingly. Dec 10 '20 at 14:58
• You can interpret gradient boosting as gradient descent in the functional space. The residual vector of each subsequent model sets the gradient vector. I'll refer you to this excellent 3-part explanation: explained.ai/gradient-boosting/descent.html#alg:general Dec 10 '20 at 15:02