7
$\begingroup$

I want to extract each tree so that I can feed it with any data, and see the output.

dump_list=xg_clas.get_booster().get_dump()
num_t=len(dump_list)
print("Number of Trees=",num_t)

I can find number of trees like this,

xgb.plot_tree(xg_clas, num_trees=0)
plt.rcParams['figure.figsize']=[50, 10]
plt.show()

graph each tree like this. When I do something like:

dump_list[0]

it gives me the tree as a text. But I couldn't find any way to extract a tree as an object, and use it.

https://github.com/dmlc/xgboost/issues/117#ref-commit-3f6ff43 I found this but didn't really understand what is suggested.

Progress: I tried to somehow turn

dump_list[0]

string object into a sklearn DecisionTreeClassifier object. Still no luck.

I uploaded my notebook if you want to check it out: https://github.com/sciencelove11/Question

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ As indicated in the answer to your last question (datascience.stackexchange.com/a/57874/55122), in gradient boosted trees the individual tree's outputs are not 0/1, but adjustments to the previous (additive) scores (which are generally approximations of the log-odds, not the probabilities). Those values are printed in the leaves in the plot_tree method. $\endgroup$
    – Ben Reiniger ♦
    Aug 20, 2019 at 20:47
  • 1
    $\begingroup$ @BenReiniger You are right, what I want is extract each tree and feed with the data that I like. To find output of each individual tree according to my data. I supposed to be more clear, I will edit my post. $\endgroup$
    – J.Smith
    Aug 20, 2019 at 21:18

1 Answer 1

Reset to default
7
+50
$\begingroup$

This is an open feature request (at time of writing):
https://github.com/dmlc/xgboost/issues/2175
https://github.com/dmlc/xgboost/issues/3439
There, a very wasteful but working solution is mentioned: predict using ntree_limit for each number of trees of interest. I've put together a quick demonstration Colab notebook here.

It also has been asked several times over at SO, see e.g.
https://stackoverflow.com/questions/51681714/extract-trees-and-weights-from-trained-xgboost-model
https://stackoverflow.com/questions/37677496/how-to-get-access-of-individual-trees-of-a-xgboost-model-in-python-r
and their Related questions.
In the first link, another workaround is mentioned: by dumping to text/PMML, you should be able to reload each individual tree (or subsets thereof) and make the predictions. It's not clear how to make this work though: XGB itself doesn't have an easy way to load a model except from its own binary format. You might be able to do it by parsing the output (JSON seems most promising) into another library with tree models.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Hello. I already saw the last SO question, and implied. First SO question redirects to another question which doesn't exist anymore. At first Github link it suggests to do >xgboost.Booster.predict() . I do it like a=xgb().Booster.predict(data=cancer.data) and get 'modue' object is not callable error, when I do from xgboost import Booster and call the same line with a=Booster.predict(data=cancer.data) I get missing 1 required positional argument: 'self' error. I already have the tree as string, I can't convert it to object. And I want specific tree output, ntree_limit just limits it. $\endgroup$
    – J.Smith
    Aug 22, 2019 at 21:37
  • 1
    $\begingroup$ @J.Smith In the first suggestion, you should be calling xg_clas.predict(...) with each value of ntree_limit, then take the difference of successive predictions. (Probably you should work in the log-odds space; I think that amounts to using output_margin=True?) And yeah, I can't quickly find a way to use the get_dump text: there's stackoverflow.com/a/44114365/10495893 that points out a package that appears to make use of that text format, and you should be able to export as JSON which might be easier to work with manually or find another package? $\endgroup$
    – Ben Reiniger ♦
    Aug 23, 2019 at 2:59
  • $\begingroup$ Hello, xg_clas.predict(ntree_limit=N) just limits the number of trees in the prediction xgboost.readthedocs.io/en/latest/python/python_api.html. I didn't understand what you meant by successive predictions. You mean I make ntree_limit=N and ntree_limit=N-1 subtract and find the prediction at tree N? I don't think it will work that way, they are not mathematical objects. I checked the link you sent, I tried to put text of tree in notepad and open it. I encounter with error, I think the problem here is you have to save the "object" first. What we do is save a string. $\endgroup$
    – J.Smith
    Aug 23, 2019 at 4:08
  • 1
    $\begingroup$ I uploaded my notebook at my edited post. Maybe it would be more clear after seeing the code. $\endgroup$
    – J.Smith
    Aug 23, 2019 at 4:21
  • 1
    $\begingroup$ The ntree_limit approach seems to work. github.com/bmreiniger/datascience.stackexchange/blob/master/… $\endgroup$
    – Ben Reiniger ♦
    Aug 23, 2019 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.