I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) sequentially -- each subsequent one is learning from the errors of the previous ones (by upweighting the misclassified examples, if I see correctly) and combining the results.

(Subquestion: does this combination mean that we use all the subsequently trained constituent estimators, maybe with different weights, or we just take the final one, which is assumed to be the most accurate?).

However, I cannot figure out how gradient descent and learning rate comes into the picture here. Trees themselves are not gradient based learners, and combining the output (either way) doesn't require any optimization. So what is its role?

  • $\begingroup$ It seems you have mixed Adaptive Boosting and Gradient Boosting. Please read this or any other reference and update the question accordingly. $\endgroup$ – 10xAI Dec 10 '20 at 14:58
  • 1
    $\begingroup$ You can interpret gradient boosting as gradient descent in the functional space. The residual vector of each subsequent model sets the gradient vector. I'll refer you to this excellent 3-part explanation: explained.ai/gradient-boosting/descent.html#alg:general $\endgroup$ – Akylas Stratigakos Dec 10 '20 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.