1
$\begingroup$

While passing the exact same parameters to LightGBM and sklearn's implementation of LightGBM, I am getting different results. Initially, I was getting the exact same results on doing this, however, I made some changes to my code and now I can't find out why they're not coming the same. This means that the performance metrics and feature importance are coming differently. Please help me figure it out, I can't figure out the mistake I am making. It could either be a mistake in the way I am implementing LightGBM using the original library or in sklearn's implementation. Link for explanation on why we should get identical results - https://stackoverflow.com/questions/51077816/light-gbm-python-api-vs-scikit-learn-api/51077977#51077977

x_train, x_test, y_train, y_test = train_test_split(df_dummy[df_merge.columns], labels, test_size=0.25,random_state=42)

n_folds = 5

lgb_train = lgb.Dataset(x_train, y_train)

def objective(params, n_folds = n_folds):
    """Objective function for Gradient Boosting Machine Hyperparameter Tuning"""

    print(params)

    params['max_depth'] = int(params['max_depth'])
    params['num_leaves'] = int(params['num_leaves'])

    params['min_child_samples'] = int(params['min_child_samples'])
    params['subsample_freq'] = int(params['subsample_freq'])

    # Perform n_fold cross validation with hyperparameters

    # Use early stopping and evalute based on ROC AUC
    cv_results = lgb.cv(params, lgb_train, nfold=n_folds, num_boost_round=10000, 
                        early_stopping_rounds=100, metrics='auc')

    # Extract the best score
    best_score = max(cv_results['auc-mean'])

    # Loss must be minimized
    loss = 1 - best_score
    num_iteration = int(np.argmax(cv_results['auc-mean']) + 1)

    of_connection = open(out_file, 'a')
    writer = csv.writer(of_connection)
    writer.writerow([loss, params, num_iteration])

    # Dictionary with information for evaluation
    return {'loss': loss, 'params': params, 'status': STATUS_OK, 'estimators': num_iteration}

space = {
    'min_child_samples': hp.quniform('min_child_samples', 5, 100, 5), 
    'reg_alpha': hp.uniform('reg_alpha', 0.0, 1.0),
    'reg_lambda': hp.uniform('reg_lambda', 0.0, 1.0),
    'colsample_bytree': hp.uniform('colsample_bytree', 0.5, 1.0),
    'max_depth' : hp.quniform('max_depth', 3, 10, 1),
    'subsample' : hp.quniform('subsample', 0.6, 1, 0.05),
    'num_leaves': hp.quniform('num_leaves', 20, 150, 1),  
    'subsample_freq': hp.quniform('subsample_freq',0,10,1),
    'min_gain_to_split': hp.quniform('min_gain_to_split', 0.01, 0.1, 0.01),


    'learning_rate' : 0.05,
    'objective' : 'binary',

}

out_file = 'results/gbm_trials.csv'
of_connection = open(out_file, 'w')
writer = csv.writer(of_connection)

writer.writerow(['loss', 'params', 'estimators'])
of_connection.close()

trials = Trials()
best = fmin(objective, space, algo=tpe.suggest, trials=trials, max_evals=10)
bayes_trials_results = sorted(trials.results, key = lambda x: x['loss'])

results = pd.read_csv('results/gbm_trials.csv')

# Sort with best scores on top and reset index for slicing
results.sort_values('loss', ascending = True, inplace = True)
results.reset_index(inplace = True, drop = True)
results.head()
best_bayes_estimators = int(results.loc[0, 'estimators'])

best['max_depth'] = int(best['max_depth'])
best['num_leaves'] = int(best['num_leaves'])

best['min_child_samples'] = int(best['min_child_samples'])

num_boost_round=int(best_bayes_estimators * 1.1)
best['objective'] = 'binary'
best['boosting_type'] = 'gbdt'

best['subsample_freq'] = int(best['subsample_freq'])

#Actual LightGBM

best_gbm = lgb.train(params=best, train_set=lgb_train, num_boost_round=num_boost_round)

#Sklearn's Implementation of LightGBM

best_sk = dict(best)
del best_sk['min_gain_to_split']
sk_best_gbm = lgb.LGBMClassifier(**best_sk, n_estimators=num_boost_round, learning_rate=0.05, min_split_gain=best['min_gain_to_split'])
sk_best_gbm.fit(x_train, y_train)

sk_best_gbm.get_params()
$\endgroup$
  • 2
    $\begingroup$ could you trim your code to the parts that are essential to the question? I think that might be helpful for folks $\endgroup$ – oW_ Oct 17 '19 at 21:01
  • $\begingroup$ Cross-posted at stackoverflow.com/q/58438962/10495893 $\endgroup$ – Ben Reiniger Oct 17 '19 at 22:06
  • $\begingroup$ I might be overlooking it (+1 to oW_'s comment), but I don't see any attempt to fix a random seed; there's some randomness in GBMs, so even if you run the native lightgbm api twice in a row you'd expect to see slightly different results. $\endgroup$ – Ben Reiniger Oct 17 '19 at 22:09
  • $\begingroup$ @BenReiniger I am sorry for the cross-posting, learning how these platforms work and need a solution very soon. I have trimmed the code by removing the parts related to the performance metrics and feature importance plotting. If I run the native lightgbm api twice in a row, I get exactly the same results in the second and first run. Initially, I was getting the exact same results in sklearn's lightgbm as well as the native api, but after making a few code changes to the parameters and syntax, this is no longer happening. Thank you for your help! $\endgroup$ – Sanchez_P Oct 18 '19 at 6:06
  • $\begingroup$ @oW_ I have trimmed the code by removing the parts related to the performance metrics and feature importance plotting. $\endgroup$ – Sanchez_P Oct 18 '19 at 6:10
1
$\begingroup$

A kind soul from Github was able to answer this - https://github.com/microsoft/LightGBM/issues/2517

The answer is this: for the parameters consistency problem, you can try to use a new lgb_train before lgb.train, like

lgb_train = lgb.Dataset(x_train, y_train)
best_gbm = lgb.train(params=best, train_set=lgb_train, num_boost_round=num_boost_round)

lgb_train is lazy-inited, and only inited one-time, so it will be constructed in the cv part. And some parameters (like min_child_samples) in that part may change the lgb_train. Therefore, lgb_train may is inited by different parameters. (So it is better to use a new lgb.train in cv part as well.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.