Small number of estimators in gradient boosting

Question

I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models and Bayesian optimization algorithm for the hyperparameters search (hyperopt)

One of the hyperparameter which is being tuned in the number of estimators used for each model. For the 1st round of testing, I starting with number of estimators in the range of 100-300. The outcome from hyperparameters optimization algorithm was that the best number of estimators is 100.I repeated the same analysis where the range of number of estimators was modified to 50-300, this time the outcome from hyperparameters optimization algorithm was that the best number of estimators is 50. I repeated the same analysis for the 3rd time setting the range of possible number of estimators to 2-300 (Just to check the extreme case) this time the outcome from hyperparameters optimization algorithm was that the best number of estimators is 2. The same outcome was noticed both for the XGBoost and lightGBM models.

What does this say about my model and data? Does this mean that the best model is Random forest instead of gradient boosting? is it smart to 'force' the number of estimators by fixing the hyperparameter value (e.g., 150) and tuning all other parameters?

The training dataset has > 0.5M Samples where ~ 90% of the data is zero and the rest is mix of positive and negative values. Should I consider using a different object function rather than MSE?

The range of the tuned parameters for LightGBM are:

'max_depth': 5 ==> 15
'colsample_bytree': .6==> .9
'subsample':.5 ==> .8
'reg_alpha': np.log(1e-4) ==> np.log(1e-1)
'reg_lambda': np.log(1e-4) ==> np.log(1e-1)
'n_estimators': 50 ==> 300
'num_leaves': 10 ==> 150, 2
'min_child_samples': 20 ==> 800
'subsample_for_bin': 20000 ==> 300000
'subsample_freq': 1 ==> 20

Output: {'subsample_freq': 16, 'num_leaves': 10, 'max_depth': 6, 'colsample_bytree': 0.7577614465604802, 'subsample_for_bin': 80000, 'min_child_samples': 415, 'n_estimators': 56, 'subsample': 0.6531478473538894, 'reg_alpha': 0.025744268683186224, 'reg_lambda': 0.0001729942781329532}

The range of the tuned parameters for XGBoost are:

'colsample_bytree',.6 ==> .9
'max_depth', 4 ==> 9
'n_estimators', 50 ==> 300
'reg_alpha', np.log(1e-4) ==> np.log(1e-2)
'subsample',.5 ==> .8
'gamma', 0 ==> 4

Output: {'gamma': 2.4257700330471357, 'max_depth': 4, 'n_estimators': 57, 'subsample': 0.5568564232616263, 'reg_alpha': 0.0009876777981446033, 'colsample_bytree': 0.7073279309167877}

Answer 1

First what is n_estimators:

n_estimatorsinteger, optional (default=10) The number of trees in the forest.

Gradient Boosting and Random Forest are decision trees ensembles, meaning that they fit several trees and then they average(ensemble) them.

If you have n_estimators=1, means that you just have one tree, if you have n_estimators=3 means that you have 3 trees and that it predicts the results of each tree and then it "averages" the result to get you the best.

It is not a good idea to force hyperparameters, cross-validation is the way to go in hyperparameter selection.

If you make your search a bit bigger you might be able to find different results. Per example move wider in the subsample, in the penalties, in the max depth...

More parameters to search will mean more computational time.