In neural networks there is an idea of a "learning rate schedule" which changes the learning rate as training progresses.

This made me ask the question, what would be the impact of varying parameters in a GBM as a function of the number of trees?

Take the learning rate for example. For GBMs using the MART algorithm, the contribution of each tree is weighted by a function of the error and the learning rate. Trees fit early on have a higher impact; trees fit later on have less impact. What if the learning rate was a function of $N$ such as $\exp(-a N)$ where $a$ would be the decay parameter of the learning rate?

Other parameters could vary as well. For example the max depth of each tree could start out high and then decrease as training progresses. Going beyond just the tree parameters, other examples are the subsample percentage if using bagging or parameters of a loss function (e.g., Huber loss parameter $\delta$).


Learning rate decay is implemented in, e.g., XGBoost and LightGBM as callbacks. (XGBoost used to allow the learning_rate parameter to be a list, but that was deprecated in favor of callbacks.) Similar functionality for other hyperparameters should be possible in the same way.

LightGBM callback discussion

I played around with these ideas (for learning rate and tree depth) a while back, but didn't get improved performance. But you should try it out; if you do see significant gains, it'd be great to add it as an answer here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.