Recommendations for tuning XGBoost Hyperparams?

Question

XGBoost has quite a few hyperparameters to tune: max depth, min child weight, number of iterations, eta, gamma, percent of columns considered, and percent of samples considered.

It's computationally infeasible to tune all of these simultaneously in a huge grid search. So, these must be done in some order.

Do you have any recommendations?

Currently, I first tune Eta and N iterations together, then Max Depth and MCW together, then col-sample and row-sample together, then finally gamma.

Do you have other ideas? If you tune it piece-wise like this, how do you decide at what value to fix the hyperparams at the very start? For example, what do you set Max Depth and MCW when you're tuning Eta etc.?

Answer 1

The stepwise algorithm for XGBoost hyperparameter tuning is inspired by a similar algorithm for LightGBM explained in this post.

The most commonly used and the most effective XGBoost parameters are split into 3 groups:

GROUP 1: max_depth , min_child_weight

GROUP 2: subsample, colsample_bytree

GROUP 3: learning_rate, num_boost_round

Initially, learning_rate and num_boost_round are fixed at 0.1 and 1000 respectively.

Each of these groups of hyperparameters are tuned sequentially. While tuning a particular group, all the subsequent groups are fixed at default or initial values and all the preceding groups are fixed at the values obtained after the tuning process. For example, by the time execution reachs GROUP 2, GROUP 1 is already tuned so we will fix GROUP 1 at the optimal values obtained, while the parameters in the subsequent groups (only GROUP 3 in this case) are left default or at the intialized values (0.1 and 1000 in this case) since they still need to be tuned.

The benefit of stepwise tuning is that the hyperparameter space is narrowed down to the group being tuned. In conventional tuning methods, we tune all the hyperparameters togeather which requires searching through a larger space. For instance, in this case we have 6 hyperparameters, tuning all of them together will involve searching through a 6 dimensional space. However, if stepwise algorithm is used, we will have to search a space of only 2 dimensions at once which is way more efficient and faster than searching through a larger space.