I've been looking into Bayesian optimization for hyperparameter tuning and trying to compare the results I get to those I get using different methods (random grid search).

I came across this site, where the author uses the mlrMBO package to MAXIMIZE the log-likelihood (see Example #2): https://www.simoncoulombe.com/2019/01/bayesian/. I have a different scenario, where I want to MINIMIZE the log-loss, so I made some minor adjustments to the author's code when defining the objective function, but I am not sure if it is correct. His objective function returned the maximum value of the test log-likelihood obtained via cross-validation and the minimize argument in the makeSingleObjectiveFunction function in the smoof library is set to FALSE. Since I want to minimize the log-loss, I returned the minimum of the log-loss from cross-validation and set the minimize argument to TRUE. Because this is my first attempt at using the package and am not too savvy with machine learning in general, I am not sure if my code is right. Any insights would be greatly appreciated!

obj.fun  <- makeSingleObjectiveFunction(
  name = "xgb_cv_bayes",
  fn = function(x){
    cv <- xgb.cv(params = list(
      booster          = "gbtree",
      eta              = x["eta"],
      max_depth        = x["max_depth"],
      min_child_weight = x["min_child_weight"],
      gamma            = x["gamma"],
      subsample        = x["subsample"],
      colsample_bytree = x["colsample_bytree"],
      objective        = "binary:logistic", 
      eval_metric     = "logloss"),
      data = dtrain,
      nrounds = x["nrounds"],
      folds =  cv_folds,
      prediction = FALSE,
      showsd = TRUE,
      early_stopping_rounds = 10,
      verbose = 0)

    cv$evaluation_log[, min(test_logloss_mean)]
  par.set = makeParamSet(
    makeNumericParam("eta",              lower = 0.1, upper = 0.5),
    makeNumericParam("gamma",            lower = 0,   upper = 5),
    makeIntegerParam("max_depth",        lower = 3,   upper = 6),
    makeIntegerParam("min_child_weight", lower= 1,    upper = 2),
    makeNumericParam("subsample",        lower = 0.6, upper = 0.8),
    makeNumericParam("colsample_bytree", lower = 0.5, upper = 0.7),
    makeIntegerParam("nrounds", lower = 100, upper = 1000)
  minimize = TRUE

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.