0
$\begingroup$

I have a matrix with each row corresponds to a hyperparameter for the XGBoost model. There are seven parameters to tune in XGBoost (as shown below: nrounds/iterations, max_depth, eta, gamma, colsample_byTree, min_child_weight, and subsample). I did a literature review to specify the range and interval of values for each parameter. Using those ranges and intervals, the parameter space generated around 62,500 parameter combinations. I am using R caret::train function to generate the best hyperparameter combination for my dataset. However, the amount of simulations (62,500) is too much. I read about the Latin hypercube sampling (LHS) and I think that is what I need to reduce the number of simulations by applying initial selection of hyperparameters using LHS. But I am having trouble implementing the approach in my dataset. My goal is to generate a manageable number of hyperparameter combinations (i.e., ~500) using LHS, and then use caret::train function to select best parameters. I would like to ask for help in implementing LHS using my parameter space.

nrounds <- seq(from = 200, to = 1000, by = 200) 
maxdepth <- seq(from = 2, to = 10, by = 2)
eta <- c(0.01, 0.05, 0.1, 0.2, 0.3)
gamma <- seq(from = 0, to = 0.4, by = 0.1)
colsample_bytree <- seq(from = 0.4, to = 1, by = 0.2)
min_child_weight <- seq(from = 1, to = 5, by = 1)
subsample <- seq(from = 0.6, to = 1, by = 0.1)
dataGrid <- expand.grid(nrounds, maxdepth, eta, gamma, colsample_bytree, min_child_weight, subsample)
$\endgroup$
0
$\begingroup$

dials from Tidymodels has a grid_latin_hypercube function you can use for this https://dials.tidymodels.org/reference/grid_max_entropy.html

$\endgroup$
0
$\begingroup$
library(tidymodels)
    
xgboost_set <- param_set(list(learn_rate(range = c(0.01,0.3), trans = NULL),
                             trees(range = c(200,1000), trans = NULL), #trees(): The number of trees contained in a random forest or boosted ensemble. In the latter case, this is equal to the number of boosting iterations
                             loss_reduction(range = c(0,0.4), trans = NULL), #This corresponds to gamma in xgboost
                             tree_depth(range = c(2,10), trans = NULL),
                             min_n(range = c(1,5), trans = NULL), # assume is same with min_child_weight parameter in boosting trees
                             sample_prop(range = c(0.4,1), trans = NULL) # assume is same with min_child_weight parameter in boosting trees
                             ))
        
        # regularization_factor(range = c(0,0.4), trans = NULL),
        set.seed(463)
        me_grid <- grid_max_entropy(xgboost_set, size = 200) %>% mutate(type = "max entropy")
        ls_grid <- grid_latin_hypercube(xgboost_set, size = 200) %>% mutate(type = "latin hypercube")
        rn_grid <- grid_random(xgboost_set, size = 200) %>% mutate(type = "random")

thanks

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.