# Tuning Gradient Boosted Classifier's hyperparametrs and balancing it

I am not sure if it is a correct stack. Maybe I should have put my question into crossvalidated.

Nevertheless, I perform following steps to tune the hyperparameters for a gradient boosting model:

1. Choose loss based on your problem at hand. I use default one - deviance
2. Pick n_estimators as large as (computationally) possible (e.g. 600).
3. Tune max_depth, learning_rate, min_samples_leaf, and max_features via grid search.
4. Increase n_estimators even more and tune learning_rate again holding the other parameters fixed.

Scikit-learn provides a convenient API for hyperparameter tuning and grid search.

Let's look at python code the code:

train_gs_X, test_gs_X, train_gs_Y, test_gs_Y = train_test_split(new_features, target, random_state=42,train_size=0.1 )
gb_grid_params = {'learning_rate': [0.1, 0.05, 0.02, 0.01],
'max_depth': [4, 6, 8],
'min_samples_leaf': [20, 50,100,150],
#'max_features': [1.0, 0.3, 0.1]
}
print(gb_grid_params)

clf = grid_search.GridSearchCV(gb_gs,
gb_grid_params,
cv=2,
scoring='roc_auc',
verbose = 3,
n_jobs=10);
clf.fit(train_gs_X, train_gs_Y);


When I obtain the parameters values I cross validate modes to check overfitting.

scores = cross_validation.cross_val_score(gb,
all_data, target,
scoring="roc_auc",
n_jobs=6,
cv=3);
"Accuracy: %0.5f (+/- %0.5f)"%(scores.mean(), scores.std())


Is my approach sufficient? Is it a correct way to tune Boosted Decision Trees hyperparameters? Do you have any idea how can I improve my tuning procedure? I know there exist method like Gaussian Process, which is faster, I mean can find the optimal hyperparameters configuration in less steps but it is not issue. I want to increase the performance measured as a ROC auc.

The second issue is how to deal with unbalanced trees?

I have two ideas:

1. Use same number of signal and background (or whatever you call it) events. The problem with this approach is to skipp huge number of possibly useful events.
2. Use DecisionTree parameters class_weight

See my code below:

signal_event_no = counts = data[target == 1].count()[0]
background_event_no = counts = data[target == 0].count()[0]
ratio_background_to_signal = float(background_event_no)/signal_event_no
ratio_background_to_signal = numpy.round(ratio_background_to_signal, 3)
train_X, test_X, train_Y, test_Y = train_test_split(new_features, target, random_state=42,train_size=0.5 )