# Hyperparameter Tuning using Bayesian Techniques

I've been looking into Bayesian optimization for hyperparameter tuning and trying to compare the results I get to those I get using different methods (random grid search).

I came across this site, where the author uses the mlrMBO package to MAXIMIZE the log-likelihood (see Example #2): https://www.simoncoulombe.com/2019/01/bayesian/. I have a different scenario, where I want to MINIMIZE the log-loss, so I made some minor adjustments to the author's code when defining the objective function, but I am not sure if it is correct. His objective function returned the maximum value of the test log-likelihood obtained via cross-validation and the minimize argument in the makeSingleObjectiveFunction function in the smoof library is set to FALSE. Since I want to minimize the log-loss, I returned the minimum of the log-loss from cross-validation and set the minimize argument to TRUE. Because this is my first attempt at using the package and am not too savvy with machine learning in general, I am not sure if my code is right. Any insights would be greatly appreciated!


obj.fun  <- makeSingleObjectiveFunction(
name = "xgb_cv_bayes",
fn = function(x){
set.seed(12345)
cv <- xgb.cv(params = list(
booster          = "gbtree",
eta              = x["eta"],
max_depth        = x["max_depth"],
min_child_weight = x["min_child_weight"],
gamma            = x["gamma"],
subsample        = x["subsample"],
colsample_bytree = x["colsample_bytree"],
objective        = "binary:logistic",
eval_metric     = "logloss"),
data = dtrain,
nrounds = x["nrounds"],
folds =  cv_folds,
prediction = FALSE,
showsd = TRUE,
early_stopping_rounds = 10,
verbose = 0)

cv\$evaluation_log[, min(test_logloss_mean)]
},
par.set = makeParamSet(
makeNumericParam("eta",              lower = 0.1, upper = 0.5),
makeNumericParam("gamma",            lower = 0,   upper = 5),
makeIntegerParam("max_depth",        lower = 3,   upper = 6),
makeIntegerParam("min_child_weight", lower= 1,    upper = 2),
makeNumericParam("subsample",        lower = 0.6, upper = 0.8),
makeNumericParam("colsample_bytree", lower = 0.5, upper = 0.7),
makeIntegerParam("nrounds", lower = 100, upper = 1000)
),
minimize = TRUE
)