I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated.

Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model:

  1. Choose loss based on your problem at hand. I use default one - deviance
  2. Pick n_estimators as large as (computationally) possible (e.g. 600).
  3. Tune max_depth, learning_rate, min_samples_leaf, and max_features via grid search.
  4. Increase n_estimators even more and tune learning_rate again holding the other parameters fixed.

    Scikit-learn provides a convenient API for hyperparameter tuning and grid search.

Let's look at python code the code:

train_gs_X, test_gs_X, train_gs_Y, test_gs_Y = train_test_split(new_features, target, random_state=42,train_size=0.1 )
gb_grid_params = {'learning_rate': [0.1, 0.05, 0.02, 0.01],
              'max_depth': [4, 6, 8],
              'min_samples_leaf': [20, 50,100,150],
              #'max_features': [1.0, 0.3, 0.1] 

gb_gs = GradientBoostingClassifier(n_estimators = 600)

clf = grid_search.GridSearchCV(gb_gs,
                               verbose = 3, 
clf.fit(train_gs_X, train_gs_Y);

When I obtain the parameters values I cross validate modes to check overfitting.

scores = cross_validation.cross_val_score(gb,
                                          all_data, target,
"Accuracy: %0.5f (+/- %0.5f)"%(scores.mean(), scores.std())

Is my approach sufficient? Is it a correct way to tune Boosted Decision Trees hyperparameters? Do you have any idea how can I improve my tuning procedure? I know there exist method like Gaussian Process, which is faster, I mean can find the optimal hyperparameters configuration in less steps but it is not issue. I want to increase the performance measured as a ROC auc.

The second issue is how to deal with unbalanced trees?

I have two ideas:

  1. Use same number of signal and background (or whatever you call it) events. The problem with this approach is to skipp huge number of possibly useful events.
  2. Use DecisionTree parameters class_weight

See my code below:

signal_event_no = counts = data[target == 1].count()[0]
background_event_no = counts = data[target == 0].count()[0]
ratio_background_to_signal = float(background_event_no)/signal_event_no
ratio_background_to_signal = numpy.round(ratio_background_to_signal, 3)
train_X, test_X, train_Y, test_Y = train_test_split(new_features, target, random_state=42,train_size=0.5 )              
gb6 = GradientBoostingClassifier( n_estimators=400, learning_rate=0.2,
   class_weight=ratio_background_to_signal, max_depth=6)

Any other ideas?

The last but not the least. How can I change the hyperparameters tuning procedure in respect to xgboost? What hyperparrameters should I take care? Is it the same set as for Gradient Boosted classifier?

  • $\begingroup$ Your training set is a bit small with 10%. Is that on purpose? Are you testing the tuned parameters on the test set to check for overfitting etc.? $\endgroup$ – oW_ Oct 5 '16 at 18:37
  • $\begingroup$ Yes this reduction is done for purpose I want to increase the training time. After I obtain the best parameters values I crossvalidate (using 10 folds) with all of available data. $\endgroup$ – user1877600 Oct 5 '16 at 18:52
  • $\begingroup$ you should update your question so it shows all steps involved $\endgroup$ – oW_ Oct 5 '16 at 19:03
  • $\begingroup$ Ok done. Any comments on my approach? $\endgroup$ – user1877600 Oct 5 '16 at 19:21
  • 2
    $\begingroup$ here is an example on how to tune the parameters. the main steps are: 1. fix a high learning rate, 2.determine the optimal number of trees, 3. tune tree-specific parameters, 4. lower learning rate and increase number of trees proportionally for more robust estimators. $\endgroup$ – oW_ Oct 5 '16 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.