0
$\begingroup$

We are currently trying to figure out how LGBM creates its trees and how predictions are made afterwards.

In my current understanding, it works as follows:

  • Multiple "weak learners" are created sequentially by taking a subset of the data, building a decision tree, taking another subset of the data (where formerly wrongly classified datapoints are picked more likely), building a decision tree, and so on.
  • Once a certain number of decision trees are built, the ensemble can be used for a prediction. For this, a datapoint runs through each individual decision tree to create an individual prediction. Afterwards, a mayority vote takes place (which class is predicted most often?) to determine the final prediction of the ensemble. In case of regression, the predictions are somehow aggregated, e.g. by taking the mean.

Is this correct?

For the prediction, there is also the conception that each datapoint runs through the trees sequentially whereas each tree changes the prediction of the last one. Which explaination is correct; this one or the one above?

Or, more visually, is A or B correct? enter image description here

Thanks a lot for clearing this up!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Your explanation is closer to how Random Forest works. It builds many weak parallel models, and then concatenates them.

Gradient Boosting methods create the weak learners sequentially, and use gradient optimization to make each "generation" better, since it tries to also predict what the previous generation got wrong. Then, the final prediction is produced from the sum of those learners outputs, not each of them sequentially.

In other words, Image B is closer to the truth (in some way), but only for the training part. When making a prediction, it is more similar to Image A.

I am not that much familiar with LGBM specifically, so I'll leave it to someone else to explain its specifics.

Improve this answer
$\endgroup$
6
  • $\begingroup$ "When doing a prediction, it gets fed directly into the last tree." is not correct; the ensemble is still additive. (Consider boosted stumps: you wouldn't want just two possible output values.) $\endgroup$
    – Ben Reiniger ♦
    Jul 11 at 12:55
  • $\begingroup$ I didn't know that. Can you point me to some resources to read more? $\endgroup$
    – liakoyras
    Jul 11 at 14:15
  • $\begingroup$ The xgboost introduction is pretty good. I really like the sklearn User Guide generally; here's their GBM section. $\endgroup$
    – Ben Reiniger ♦
    Jul 16 at 1:36
  • $\begingroup$ Thanks. I have read that sklearn page so many times over the years and yet I still somehow didn't register that part. I tried to edit my answer to correct this. $\endgroup$
    – liakoyras
    Jul 17 at 6:10
  • $\begingroup$ I still disagree with the part "When making a prediction, it is more similar to Image A." Making predictions happens exactly as in image B. What neither image makes clear is during training, how the weak learners are fit. Your answer does partially address that, but could emphasize that the Question's first bullet is closer to AdaBoost, not gradient boosting. $\endgroup$
    – Ben Reiniger ♦
    Nov 8 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.