4
$\begingroup$

For Gradient Boosting Models such as XGBOOST and LGBM does n_warmup_steps in optuna.pruners.MedianPruner refer to the minimum number of folds evaluated before pruning is triggered?

I.e. if number of CV folds equals 5 then n_warmup_steps=1 means pruning only takes place after at least one 2/5 folds is evaluated (since it starts counting at 0)?

$\endgroup$

1 Answer 1

4
$\begingroup$

The steps in n_warmup_steps refer to the incremental steps taken during gradient decent. So with n_warmup_steps=1 the trial will directly check the loss for the very first incremental step. To put it differently, with n_warmup_steps=1 a trial has to achieve a good result immediatly after the gradient decent started.
As you are using optuna.pruners.MedianPruner following applies (copied from the official documentation):

Prune if the trial’s best intermediate result is worse than median of intermediate results of previous trials at the same step.

So if a trial does not yield a better result (measured via the objective/loss function) than the median of the intermediate results at the same step, it will be pruned.
The parameter n_warmup_steps can be used in conjunction with n_startup_trials in order to get a robust median value first - so numbers of trials without pruning. After n_startup_trials, the trials will be pruned after n_warmup_steps if they performed worse than the preceding trials after the same amount of steps while gradient decending.

The rationale behind the parameter n_warmup_steps ist that a trial can start with a loss value below the median, but can give a much better loss value eventually - after more iterations (e.g. depending on the learning rate [...]).

It's not easy for me to explain it, maybe the images at the beginning of this article give a better intuition.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.