I have been searching for a while and I just can't find any indication. When people talk about iterations in algorithms like XGBoost or LightGBM, or Catboost, do they mean how many decision trees i.e. base learners will be built? I.e. XGboost m=100 means the algorithm will build a total of 100 base learners, each calculating and optimizing towards the residual value of the previous prediction?
Or is it more like 1 epoch in deep learning?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Your first interpretation is correct. One base learner will be added per boosting iteration/round and that is probably what people are referring to when talking about iterations.
From wiki:
One natural regularization parameter is the number of gradient boosting iterations M (i.e. the number of trees in the model when the base learner is a decision tree).
Iterations take place in other parts of the algorithm, for instance in the gradient descent, but I don't think that is what is discussed if it is simply referred to as "iterations".
This was really helpful when I was trying to understand GBMs
answered May 29, 2019 at 16:36