We are currently trying to figure out how LGBM creates its trees and how predictions are made afterwards.
In my current understanding, it works as follows:
- Multiple "weak learners" are created sequentially by taking a subset of the data, building a decision tree, taking another subset of the data (where formerly wrongly classified datapoints are picked more likely), building a decision tree, and so on.
- Once a certain number of decision trees are built, the ensemble can be used for a prediction. For this, a datapoint runs through each individual decision tree to create an individual prediction. Afterwards, a mayority vote takes place (which class is predicted most often?) to determine the final prediction of the ensemble. In case of regression, the predictions are somehow aggregated, e.g. by taking the mean.
Is this correct?
For the prediction, there is also the conception that each datapoint runs through the trees sequentially whereas each tree changes the prediction of the last one. Which explaination is correct; this one or the one above?
Or, more visually, is A or B correct?
Thanks a lot for clearing this up!