I fail to understand as to how learning rate is used in XGBoost? Can anyone explain using a numerical example?
Each iteration is supposed to provide an improvement to the training loss. Such improvement is multiplied with the learning rate in order to perform smaller updates.
Smaller updates allow to overfit slower the data, but requires more iterations for training.
For instance, doing 5 iteations at a learning rate of 0.1 approximately would require doing 5000 iterations at a learning rate of 0.001, which might be obnoxious for large datasets.
Typically, we use a learning rate of 0.05 or lower for training, while a learning rate of 0.10 or larger is used for tinkering the hyperparameters...
Gradient boosting algorithms are ensembling algorithms, where many classifiers vote in a bigger classifier. In the case of gradient boosting, trees being trained on the residuals of the previous one are the weak classifiers that vote. They don't do purely democratic voting, as their votes are weighted. The learning rate is the rate of the wieghts of the votes between a weak classifier and the next one.