Dynamic learning rates in XGBoost cross-validation

Question

XGBoost's xgb.train() method takes a learning_rates parameter, which can take a custom function to apply a dynamic learning rate, depending on the current training round.

I recently posted a paper explaining how I'm using it to both speed up training in the beginning, and making more precise towards the end.

However, there's a problem with this method: it tends to overfit on the eval set because there's no cross-validation.

I noticed that xgb.cv() method has no learning_rates parameter and therefore appears to not allow for a dynamic learning rate while doing CV.

So my questions:

• Is there a possibility to use dynamic learning rates with xgb.cv()?
• If not, is there a rationale behind this or is this just not implemented yet?

Answer 1

There isn't as far as I can tell by looking at the source code of the version I'm working with (April 2016). But it's definitely worth adding and it isn't impossible. xgboost.train loops over the iterations extracts corresponding learning rate, sets it on booster via eta parameter and calls booster update. xgboost.cv calls fold update (ie CVPack.update) which calls booster update. Just need to add a learning rate (aka eta) parameter to CVPack.update