0
$\begingroup$

XGBoost's xgb.train() method takes a learning_rates parameter, which can take a custom function to apply a dynamic learning rate, depending on the current training round.

I recently posted a paper explaining how I'm using it to both speed up training in the beginning, and making more precise towards the end.

However, there's a problem with this method: it tends to overfit on the eval set because there's no cross-validation.

I noticed that xgb.cv() method has no learning_rates parameter and therefore appears to not allow for a dynamic learning rate while doing CV.

So my questions:

  • Is there a possibility to use dynamic learning rates with xgb.cv()?
  • If not, is there a rationale behind this or is this just not implemented yet?
$\endgroup$
  • $\begingroup$ Have you tried setting in the eta Hyper-Parameter, responsible for the learning rate. Have a look here github. Moreover, have tried uploading your question at the dmlc XGBoost github issues page? Furthermore, regarding your approach, I would be interested to see it in more use cases, I might try it. However, Gradient Descent is a complicated thing and the fact that XGBoost can be trained in different sub-samples, keeps me wondering how consistent this methodology would be. $\endgroup$ – Grzegorz Nov 14 '16 at 14:49
1
$\begingroup$

There isn't as far as I can tell by looking at the source code of the version I'm working with (April 2016). But it's definitely worth adding and it isn't impossible. xgboost.train loops over the iterations extracts corresponding learning rate, sets it on booster via eta parameter and calls booster update. xgboost.cv calls fold update (ie CVPack.update) which calls booster update. Just need to add a learning rate (aka eta) parameter to CVPack.update

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks for the comment. Actually I came to the same conclusion and submitted a PR: github.com/dmlc/xgboost/pull/1770 - however the team seems quite busy right now and it hasn't been looked at yet $\endgroup$ – Jivan Nov 18 '16 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.