Hyperparameter tuning with Bayesian-Optimization

Question

I'm using LightGBM for the regression problem and here is my code.

    def bayesion_opt_lgbm(X, y, init_iter = 5, n_iter = 10, random_seed = 32, seed= 100, num_iterations = 50,
                          dtrain = lgb.Dataset(data = X_train, label = y_train)):
      def lgb_score(y_preds, dtrain):
        labels = dtrain.get_labels()
        return 'r2', r2_score(labels, y_preds), True
    
      def hyp_lgb(num_leaves, feature_fraction, bagging_fraction, max_depth, min_split_gain, min_child_weight):
          params = {'application': 'regression',
                    'num_iterations': 'num_iterations',
                    'early_stopping_round': 50,
                    'learning_rate': 0.05,
                    'metric': 'lgb_r2_score'}
          params['num_leaves'] = int(round(num_leaves))
          params['feature_fraction'] = max(min(feature_fraction, 1), 0)
          params['bagging_fraction'] = max(min(bagging_fraction, 1), 0)
          params['max_depth'] = int(round(max_depth))
          params['min_split_gain'] = min_split_gain
          params['min_child_weight'] = min_child_weight
          cv_results = lgb.cv(params, 
                              train_set = dtrain, 
                              nfold = 5, 
                              stratified = False,
                              seed = seed,
                              categorical_feature = [],
                              verbose_eval = None,
                              feval = lgb_r2_score)
          print(cv_results)
          return np.max(cv_results['r2-mean'])
      bounds = {'num_leaves': (80,100),
                'feature_fraction': (0.1, 0.9),
                'bagging_fraction': (0.8, 1),
                'max_depth': (5,10,15,20),
                'min_split_gain': (0.001, 0.01),
                'min_child_weight': (10,20)
                }
    
      optimizer = BayesianOptimization(f = hyp_lgb, pbounds = bounds, random_state = 32)
    
      optimizer.maximaze(init_points= init_iter, n_iter = n_iter)
    bayesion_opt_lgbm(X_train, y_train)

When I run my code, I get an error something like that, Please help me where am i missing

TypeError                                 Traceback (most recent call last)
TypeError: float() argument must be a string or a number, not 'tuple'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-57-86f7d803c78d> in <module>()
     40   #Optimize
     41   optimizer.maximaze(init_points= init_iter, n_iter = n_iter)
---> 42 bayesion_opt_lgbm(X_train, y_train)
     43 

2 frames
/usr/local/lib/python3.6/dist-packages/bayes_opt/target_space.py in __init__(self, target_func, pbounds, random_state)
     47         self._bounds = np.array(
     48             [item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])],
---> 49             dtype=np.float
     50         )
     51 

ValueError: setting an array element with a sequence.
  

Answer 1

The pbounds must all be pairs; you cannot specify a list of options for max_depth.

The package cannot deal with discrete hyperparameters very directly; see section 2, "Dealing with discrete parameters", of their "advanced tour" notebook about this.