How can the process of hypertuning of XGBoost parameters be automated?

Question

I'm using xgboost for training a model on a data with extreme class imbalance. After referring from here.

After performing grid search and some manual settings, I found that the following parameters work the best for me:

weight <- as.numeric(labels) * nrow(test) / length(labels)
upscale <- sum(weight * (labels == 1.0))

xgb_params = list(
  objective = 'binary:logistic',                                               
  eta = 0.1,                                                                  
  max_depth = 4,                                                               
  eval_metric = 'auc',                                                         
  max_delta_step = 10,
  scale_pos_weight = upscale
)

How can the process of setting optimal hyperparameters for xgboost be automated for best AUC? Please note that some of these parameters aren't supported by the caret implementation of xgboost but are very important for the model I have to design.

Answer 1

In general, if you want to automate fine tuning a model's hyper parameters, its best to use a well tested package such as caret or MLR.

I've used the caret package extensively. Here is a reference of the parameters supported by caret for tuning a xgboost model.

To automatically select parameters using caret, do the following:

1. First define a range of values of each parameter you would want caret to search. Define this in the tuning grid.
2. Start model training using caret after specifying a measure to optimize, e.g. accuracy or Kappa statistic, etc.
3. Plot or print the performance comparison for various parameter values, refine and repeat if required.

Refer to the caret guide here to get step-by-step instructions on using it.

For handling class imbalance, I've found from my experience that adjusting weights is not as helpful as under-sampling majority class and over-sampling the minority class, or a combination of the two. However, it all depends on the size of data available and the case at hand.

In case you need to tune some parameters which are not supported by caret, then, you could write your own iterative loop to train and test the model for different values of that parameter and then choose one that works best. I think most of the really relevant parameters have already been included in caret.

You would need to adjust these parameters in case the population itself changes over time. Or, the methods to gather data and their accuracy may change which could result in performance deterioration. You could run a simple check by comparing the performance of your model over the current dataset vs. a 6 month older dataset. If the performance is similar, then you may not need to update the model in the future.