I have used XGBoost to train a model with 400 features. My understanding is that since the max_depth is default at only 6, and 2^6 < 400, not all features will end up in the tree.

How come when I output the feature importance chart, it shows every single feature with above 0 importance? The decision tree output clearly shows that not every feature has been used in the final tree.


1 Answer 1


XGBoost iteratively trains many trees (=boosting). So there is not only one tree. For example you can plot a single tree, see: https://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/

A basic decision tree algorithm creates just one tree. If you apply pruning to the tree not all features would be present in the tree. The first split would be the one with the highest importance, ...

  • $\begingroup$ So when I am using XGBoost.predict, its using all the trees it made? How does it combine the outputs? $\endgroup$ Dec 21, 2020 at 22:07
  • $\begingroup$ Yes, take a look at: datascience.stackexchange.com/questions/9862/… $\endgroup$ Dec 21, 2020 at 22:17
  • $\begingroup$ But isn't that only if I set num_parallel_trees = to something? Ie. a randomforest? $\endgroup$ Dec 21, 2020 at 22:25
  • $\begingroup$ Boosting is an Ensembling Training Method which creates multiple trees. See official paper, section 2.1 - arxiv.org/pdf/1603.02754.pdf $\endgroup$ Dec 21, 2020 at 22:30
  • $\begingroup$ The first split won't necessarily be the on the feature of highest importance, depending on your definition of importance. $\endgroup$
    – Ben Reiniger
    Dec 21, 2020 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.