# What happens if GBM parameters (e.g., learning rate) vary as the training progresses?

In neural networks there is an idea of a "learning rate schedule" which changes the learning rate as training progresses.

This made me ask the question, what would be the impact of varying parameters in a GBM as a function of the number of trees?

Take the learning rate for example. For GBMs using the MART algorithm, the contribution of each tree is weighted by a function of the error and the learning rate. Trees fit early on have a higher impact; trees fit later on have less impact. What if the learning rate was a function of $$N$$ such as $$\exp(-a N)$$ where $$a$$ would be the decay parameter of the learning rate?

Other parameters could vary as well. For example the max depth of each tree could start out high and then decrease as training progresses. Going beyond just the tree parameters, other examples are the subsample percentage if using bagging or parameters of a loss function (e.g., Huber loss parameter $$\delta$$).

Learning rate decay is implemented in, e.g., XGBoost and LightGBM as callbacks. (XGBoost used to allow the learning_rate parameter to be a list, but that was deprecated in favor of callbacks.) Similar functionality for other hyperparameters should be possible in the same way.