Page 359 of Elements Of Statistical Learning 2nd edition says the below.

enter image description here

Can someone explain the intuition & simplify it in layman terms?


  1. What is the reason/intuition & math behind fitting each successive tree in GBM on the negative gradient of the loss function?
  2. Is it done to make GBM more generalization on unseen test dataset? If so how does fitting on negative gradient achieve this generalization on test data?

2 Answers 2

  1. The concept is by fitting the sequential tree by the "residual" (Practically the gradient relative to the current model) the overall process will be equivalent to Gradient Descent over the loss function. So the concept is to fit the "new" tree to the negative of the gradient as to create a minimizing step with regard to the loss function.

  2. Since the model is "weak" it cannot fit the residual perfectly. Hence the fit to the data is regularized by the weakness of the model as it can only approximate the gradient step. This is a way to regularize the overall process and avoid overfitting.


The negative gradient is used because a loss function is minimized. Moving in a "downward" is reducing the expected error.

The text is trying to explain the logic of boosting. Instead of fitting the model on all the training data at one time, train the model only a subsample of the training data at a time. Then, the model can learn from its errors on subsequent blocks of training data.

  • 1
    $\begingroup$ I don't think the concept of boosting is about sub sample of the data. It is about fitting the sequential model to the current model gradient with respect to the loss function. Sub sample of data / features is the trick for Bagging / Random Forest. $\endgroup$
    – Royi
    Commented Apr 21 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.