3
$\begingroup$

I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models and Bayesian optimization algorithm for the hyperparameters search (hyperopt)

One of the hyperparameter which is being tuned in the number of estimators used for each model. For the 1st round of testing, I starting with number of estimators in the range of 100-300. The outcome from hyperparameters optimization algorithm was that the best number of estimators is 100.I repeated the same analysis where the range of number of estimators was modified to 50-300, this time the outcome from hyperparameters optimization algorithm was that the best number of estimators is 50. I repeated the same analysis for the 3rd time setting the range of possible number of estimators to 2-300 (Just to check the extreme case) this time the outcome from hyperparameters optimization algorithm was that the best number of estimators is 2. The same outcome was noticed both for the XGBoost and lightGBM models.

What does this say about my model and data? Does this mean that the best model is Random forest instead of gradient boosting? is it smart 'force' the number of estimators by fixing the hyperparameter value (e.g., 150) and tuning all other parameters?

The training dataset has > 0.5M Samples where ~ 90% of the data is zero and the rest is mix of positive and negative values. Should I consider using a different object function rather than MSE?

The range of the tuned parameters for LightGBM are:

'max_depth': 5 ==> 15
'colsample_bytree': .6==> .9
'subsample':.5 ==> .8
'reg_alpha': np.log(1e-4) ==> np.log(1e-1)
'reg_lambda': np.log(1e-4) ==> np.log(1e-1)
'n_estimators': 50 ==> 300
'num_leaves': 10 ==> 150, 2
'min_child_samples': 20 ==> 800
'subsample_for_bin': 20000 ==> 300000
'subsample_freq': 1 ==> 20

Output: {'subsample_freq': 16, 'num_leaves': 10, 'max_depth': 6, 'colsample_bytree': 0.7577614465604802, 'subsample_for_bin': 80000, 'min_child_samples': 415, 'n_estimators': 56, 'subsample': 0.6531478473538894, 'reg_alpha': 0.025744268683186224, 'reg_lambda': 0.0001729942781329532}

The range of the tuned parameters for XGBoost are:

'colsample_bytree',.6 ==> .9
'max_depth', 4 ==> 9
'n_estimators', 50 ==> 300
'reg_alpha', np.log(1e-4) ==> np.log(1e-2)
'subsample',.5 ==> .8
'gamma', 0 ==> 4

Output: {'gamma': 2.4257700330471357, 'max_depth': 4, 'n_estimators': 57, 'subsample': 0.5568564232616263, 'reg_alpha': 0.0009876777981446033, 'colsample_bytree': 0.7073279309167877}

$\endgroup$
  • $\begingroup$ What are the ranges (and reported optimum values) for the other hyperparameters in your search? $\endgroup$ – Ben Reiniger Feb 25 at 2:26
  • $\begingroup$ Ben,I just added the range of the tuned parameters. $\endgroup$ – thereandhere1 Feb 25 at 4:19
  • 1
    $\begingroup$ Consider allowing smaller trees, smaller learning rates, and (maybe) larger regularization penalties. $\endgroup$ – Ben Reiniger Feb 25 at 14:52
1
$\begingroup$

First what is n_estimators:

n_estimatorsinteger, optional (default=10) The number of trees in the forest.

Gradient Boosting and Random Forest are decision trees ensembles, meaning that they fit several trees and then they average(ensemble) them.

If you have n_estimators=1, means that you just have one tree, if you have n_estimators=3 means that you have 3 trees and that it predicts the results of each tree and then it "averages" the result to get you the best.

It is not a good idea to force hyperparameters, cross-validation is the way to go in hyperparameter selection.

If you make your search a bit bigger you might be able to find different results. Per example move wider in the subsample, in the penalties, in the max depth...

More parameters to search will mean more computational time.

| improve this answer | |
$\endgroup$
  • $\begingroup$ While I am clear on the fact that the n_estimators dictates the number of trees built, I always perceived it as number of boosting rounds. Therefore, if the n_estimators value is 1or 2 it means that is no boosting is taking place and the model turns into a RF model. Am I correct? Does small n_estimators value indicate a problem with my data? Thanks! $\endgroup$ – thereandhere1 Feb 26 at 17:05
  • $\begingroup$ There is no problem with your data. It is just that there is no need to create more trees. It has happened to me a couple of times. Tends to happen more if your subsample is same size than the sample, and if you select all columns $\endgroup$ – Carlos Mougan Mar 15 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.