# Bayesian optimization for a Light GBM Model

I am able to successfully improve the performance of my XGBoost model through Bayesian optimization, but the best I can achieve through Bayesian optimization when using Light GBM (my preferred choice) is worse than what I was able to achieve by using it’s default hyper-parameters and following the standard early stopping approach.

When tuning via Bayesian optimization, I have been sure to include the algorithm’s default hyper-parameters in the search surface, for reference purposes.

The code below shows the RMSE from the Light GBM model with default hyper-parameters using seaborn’s diamonds dataframe as an example of my workings:

pip install bayesian-optimization

import seaborn as sns
from sklearn.model_selection import train_test_split
import lightgbm as lgb
from bayes_opt import BayesianOptimization

df["color"] = df["color"].astype('category')
df["color_cat"] = df["color"].cat.codes
df = df.drop(["color"],axis = 1)

df["cut"] = df["cut"].astype('category')
df["cut_cat"] = df["cut"].cat.codes
df = df.drop(["cut"],axis = 1)

df["clarity"] = df["clarity"].astype('category')
df["clarity_cat"] = df["clarity"].cat.codes
df = df.drop(["clarity"],axis = 1)

y = df['price']
X = df.drop(['price'], axis=1)

seed = 7
test_size = 0.3
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size,random_state=seed)

train_lgb = lgb.Dataset(X_train, y_train)
eval_lgb = lgb.Dataset(X_test, y_test, reference = train_lgb)

params = { 'objective': 'regression',
'metric': 'RMSE',
'learning_rate': 0.02}
lgb_reg = lgb.train(params, train_lgb, num_boost_round = 10000, early_stopping_rounds=50, verbose_eval = 100, valid_sets=eval_lgb)



Results

OUT:
Training until validation scores don't improve for 50 rounds.
Early stopping, best iteration is:
[1330 (n_estimators)] valid_0's rmse: 538.728


Here my attempt to implement Bayesian Optimization and the resulting RMSE values:

def modelFitter(colsampleByTree, subsample,maxDepth, num_leaves):
model = lgb.LGBMRegressor(learning_rate=0.02, n_estimators=10000, max_depth=maxDepth.astype("int32"), subsample=subsample, colsample_bytree=colsampleByTree,num_leaves=num_leaves.astype("int32"))

evalSet  = [(X_test, y_test)]
model.fit(X_train, y_train, eval_metric="rmse", eval_set=evalSet, early_stopping_rounds=50, verbose=False)

bestScore = model.best_score_[list(model.best_score_.keys())[0]]['rmse']

return -bestScore

# Bounded region of parameter space
pbounds = {'colsampleByTree': (0.8,1.0), 'subsample': (0.8,1.0), 'maxDepth': (2,5), 'num_leaves': (24, 45)}

optimizer = BayesianOptimization(
f=modelFitter,
pbounds=pbounds,
random_state=1)

optimizer.maximize(init_points=5,n_iter=5)  #n_iter=bayesian, init_points=random


Results

ter    |  target   | colsam... | maxDepth  | num_le... | subsample |
-------------------------------------------------------------------------
|  1        | -548.7    |  0.8834   |  4.161    |  24.0     |  0.8605   |
|  2        | -642.4    |  0.8294   |  2.277    |  27.91    |  0.8691   |
|  3        | -583.5    |  0.8794   |  3.616    |  32.8     |  0.937    |
|  4        | -548.7    |  0.8409   |  4.634    |  24.58    |  0.9341   |
|  5        | -583.5    |  0.8835   |  3.676    |  26.95    |  0.8396   |
|  6        | -548.7    |  0.8625   |  4.395    |  24.29    |  0.8968   |
|  7        | -548.7    |  0.8435   |  4.603    |  24.42    |  0.9298   |
|  8        | -551.5    |  0.9271   |  4.266    |  24.11    |  0.8035   |
|  9        | -548.7    |  0.8      |  4.11     |  24.08    |  1.0      |
|  10       | -548.7    |  0.8      |  4.44     |  24.45    |  0.9924   |


The RMSE (-1 x “target”) generated during Bayesian optimization should be betterthan that generated by the default values of Light GBM but I cannot achieve a better RMSE (looking for better/higher than -538.728 achieved through the above mentioned “normal” early stopping process).

PS the maxDepth and num_leaves should be integers, it looks like there is an open ticket to enforce this (i.e. bringing in “ptypes”): https://github.com/fmfn/BayesianOptimization/pull/131/files

Is there a reason why the Bayesian optimization does seem to find a better solution with LightGBM but it does with XGBoost?