2
$\begingroup$

I'm doing a project. I have a classification problem that I should solve using gradient boosted decision trees. What I want to do is create a matrix that gives prediction of each decision tree for each sample. For example if I have 100 samples and 100 trees, I should have 100x100 matrix. i, j th entry gives the prediction of jth tree for ith sample.

I'm using sklearn and problem is I can't get prediction by each tree.

So far I tried:

newgb=gb.estimators_[0][0].fit(X_train, y_train)
print(newgb.score(X_train, y_train))

where gb is already a fitted model. What I understood from documentation of sklearn

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor.staged_predict

.estimators_

should return (number-of-trees x 1) matrix, each entry contains a tree that used by our model. By gb.estimators_[0][0] I tried to access to the first tree, and predict it with score. What I get as output is:

[0.12048193 0.95       0.95       0.95       0.95       0.95
 0.95       0.95       0.95       0.95       0.12048193 0.95
 0.95       0.95       0.12048193 0.12048193 0.12048193 0.12048193
...]

None of them are 1 or 0, like it should be(it is binary classification) and values repeat themselves like 0.95 and 0.12. I didn't use any likelihood function either so

.score()

supposed to give me only 1's and 0's.

I don't know how to get predictions for each individual tree. I don't know what I do wrong either.

$\endgroup$
  • $\begingroup$ Regressor or classifier? - the link in your question is to GBregressor.. $\endgroup$ – yoav_aaa Aug 20 at 12:35
2
$\begingroup$

Sklearn's GradientBoostingClassifier is not implemented using trees of DecisionTreeClassifiers. It uses regressors for both classification and regression. You can read it here:

GB builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage n_classes_ regression trees are fit on the negative gradient of the binomial or multinomial deviance loss function. Binary classification is a special case where only a single regression tree is induced.

This means it will not be as simple as calling predict on the tree estimators. I previously suggested it could be implemented using sklearn's private methods, but as BenReiniger pointed out sklearn has already implemented this for us in the method staged_predict.

$\endgroup$
  • $\begingroup$ I started with stackoverflow.com/questions/20615750/… At that link they claim it is doable. I'm a little confused. If sklearn not implemented using trees why should I look its implementation also? $\endgroup$ – J.Smith Aug 20 at 14:44
  • $\begingroup$ Sorry if it was not clear. It does use trees, but it uses DecisionTreeRegressors (predicts loss) and not DecisionTreeClassifiers (that predicts a class). The sections of code I pointed to is how sklearn goes from loss to a class. $\endgroup$ – Simon Larsson Aug 20 at 15:33
  • 1
    $\begingroup$ The ticket you linked uses random forest, which works like you have tried to do it. Boosting is fundamentally different where each tree is dependent on the tree that came before it. Maybe this can help: blog.kaggle.com/2017/01/23/… $\endgroup$ – Simon Larsson Aug 20 at 15:38
  • $\begingroup$ sorry for confusion, I'm just tired couldn't realize $\endgroup$ – J.Smith Aug 20 at 19:24
  • 2
    $\begingroup$ @J.Smith The predict_stages appears to be a more private method; the public staged_predict and staged_predict_proba call it: scikit-learn.org/stable/modules/generated/… $\endgroup$ – Ben Reiniger Aug 21 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.