# How to extract trees in XGBoost?

I want to extract each tree so that I can feed it with any data, and see the output.

dump_list=xg_clas.get_booster().get_dump()
num_t=len(dump_list)
print("Number of Trees=",num_t)


I can find number of trees like this,

xgb.plot_tree(xg_clas, num_trees=0)
plt.rcParams['figure.figsize']=[50, 10]
plt.show()


graph each tree like this. When I do something like:

dump_list[0]

it gives me the tree as a text. But I couldn't find any way to extract a tree as an object, and use it.

https://github.com/dmlc/xgboost/issues/117#ref-commit-3f6ff43 I found this but didn't really understand what is suggested.

Progress: I tried to somehow turn

dump_list[0]

string object into a sklearn DecisionTreeClassifier object. Still no luck.

I uploaded my notebook if you want to check it out: https://github.com/sciencelove11/Question

• As indicated in the answer to your last question (datascience.stackexchange.com/a/57874/55122), in gradient boosted trees the individual tree's outputs are not 0/1, but adjustments to the previous (additive) scores (which are generally approximations of the log-odds, not the probabilities). Those values are printed in the leaves in the plot_tree method. – Ben Reiniger Aug 20 '19 at 20:47
• @BenReiniger You are right, what I want is extract each tree and feed with the data that I like. To find output of each individual tree according to my data. I supposed to be more clear, I will edit my post. – J.Smith Aug 20 '19 at 21:18

This is an open feature request (at time of writing):
https://github.com/dmlc/xgboost/issues/2175
https://github.com/dmlc/xgboost/issues/3439
There, a very wasteful but working solution is mentioned: predict using ntree_limit for each number of trees of interest. I've put together a quick demonstration Colab notebook here.

It also has been asked several times over at SO, see e.g.
https://stackoverflow.com/questions/51681714/extract-trees-and-weights-from-trained-xgboost-model
https://stackoverflow.com/questions/37677496/how-to-get-access-of-individual-trees-of-a-xgboost-model-in-python-r
and their Related questions.
In the first link, another workaround is mentioned: by dumping to text/PMML, you should be able to reload each individual tree (or subsets thereof) and make the predictions. It's not clear how to make this work though: XGB itself doesn't have an easy way to load a model except from its own binary format. You might be able to do it by parsing the output (JSON seems most promising) into another library with tree models. (If I get some more time soon, I'll look into that more.)

• Hello. I already saw the last SO question, and implied. First SO question redirects to another question which doesn't exist anymore. At first Github link it suggests to do >xgboost.Booster.predict() . I do it like a=xgb().Booster.predict(data=cancer.data) and get 'modue' object is not callable error, when I do from xgboost import Booster and call the same line with a=Booster.predict(data=cancer.data) I get missing 1 required positional argument: 'self' error. I already have the tree as string, I can't convert it to object. And I want specific tree output, ntree_limit just limits it. – J.Smith Aug 22 '19 at 21:37
• @J.Smith In the first suggestion, you should be calling xg_clas.predict(...) with each value of ntree_limit, then take the difference of successive predictions. (Probably you should work in the log-odds space; I think that amounts to using output_margin=True?) And yeah, I can't quickly find a way to use the get_dump text: there's stackoverflow.com/a/44114365/10495893 that points out a package that appears to make use of that text format, and you should be able to export as JSON which might be easier to work with manually or find another package? – Ben Reiniger Aug 23 '19 at 2:59
• Hello, xg_clas.predict(ntree_limit=N) just limits the number of trees in the prediction xgboost.readthedocs.io/en/latest/python/python_api.html. I didn't understand what you meant by successive predictions. You mean I make ntree_limit=N and ntree_limit=N-1 subtract and find the prediction at tree N? I don't think it will work that way, they are not mathematical objects. I checked the link you sent, I tried to put text of tree in notepad and open it. I encounter with error, I think the problem here is you have to save the "object" first. What we do is save a string. – J.Smith Aug 23 '19 at 4:08
• I uploaded my notebook at my edited post. Maybe it would be more clear after seeing the code. – J.Smith Aug 23 '19 at 4:21
• The ntree_limit approach seems to work. github.com/bmreiniger/datascience.stackexchange/blob/master/… – Ben Reiniger Aug 23 '19 at 14:16