3
$\begingroup$

I'm using xgboost for training a model on a data with extreme class imbalance. After referring from here.

After performing grid search and some manual settings, I found that the following parameters work the best for me:

weight <- as.numeric(labels) * nrow(test) / length(labels)
upscale <- sum(weight * (labels == 1.0))

xgb_params = list(
  objective = 'binary:logistic',                                               
  eta = 0.1,                                                                  
  max_depth = 4,                                                               
  eval_metric = 'auc',                                                         
  max_delta_step = 10,
  scale_pos_weight = upscale
)

How can the process of setting optimal hyperparameters for xgboost be automated for best AUC? Please note that some of these parameters aren't supported by the caret implementation of xgboost but are very important for the model I have to design.

$\endgroup$
  • 1
    $\begingroup$ How about using a genetic algorithm to tune the parameters? $\endgroup$ – zippy Aug 4 '16 at 19:35
4
$\begingroup$

In general, if you want to automate fine tuning a model's hyper parameters, its best to use a well tested package such as caret or MLR.

I've used the caret package extensively. Here is a reference of the parameters supported by caret for tuning a xgboost model.

To automatically select parameters using caret, do the following:

  1. First define a range of values of each parameter you would want caret to search. Define this in the tuning grid.
  2. Start model training using caret after specifying a measure to optimize, e.g. accuracy or Kappa statistic, etc.
  3. Plot or print the performance comparison for various parameter values, refine and repeat if required.

Refer to the caret guide here to get step-by-step instructions on using it.

For handling class imbalance, I've found from my experience that adjusting weights is not as helpful as under-sampling majority class and over-sampling the minority class, or a combination of the two. However, it all depends on the size of data available and the case at hand.

In case you need to tune some parameters which are not supported by caret, then, you could write your own iterative loop to train and test the model for different values of that parameter and then choose one that works best. I think most of the really relevant parameters have already been included in caret.

You would need to adjust these parameters in case the population itself changes over time. Or, the methods to gather data and their accuracy may change which could result in performance deterioration. You could run a simple check by comparing the performance of your model over the current dataset vs. a 6 month older dataset. If the performance is similar, then you may not need to update the model in the future.

$\endgroup$
  • $\begingroup$ It is the parameters that caret doesn't support tuning of, that have to be tuned. Specifically, scale_pos_weight for handling imbalance. But nevermind that for now. Do you think I would have to re-adjust any of these parameters in the (near or distant) future given the nature of the data remains the same? $\endgroup$ – user140323 Jun 30 '16 at 6:24
  • $\begingroup$ Perfect! SMOTEd some of the data, much better results now! $\endgroup$ – user140323 Jun 30 '16 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.