0
$\begingroup$

I have used XGBoost to train a model with 400 features. My understanding is that since the max_depth is default at only 6, and 2^6 < 400, not all features will end up in the tree.

How come when I output the feature importance chart, it shows every single feature with above 0 importance? The decision tree output clearly shows that not every feature has been used in the final tree.

$\endgroup$
0
$\begingroup$

XGBoost iteratively trains many trees (=boosting). So there is not only one tree. For example you can plot a single tree, see: https://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/

A basic decision tree algorithm creates just one tree. If you apply pruning to the tree not all features would be present in the tree. The first split would be the one with the highest importance, ...

$\endgroup$
5
  • $\begingroup$ So when I am using XGBoost.predict, its using all the trees it made? How does it combine the outputs? $\endgroup$ – krissy_fong Dec 21 '20 at 22:07
  • $\begingroup$ Yes, take a look at: datascience.stackexchange.com/questions/9862/… $\endgroup$ – Predicted Life Dec 21 '20 at 22:17
  • $\begingroup$ But isn't that only if I set num_parallel_trees = to something? Ie. a randomforest? $\endgroup$ – krissy_fong Dec 21 '20 at 22:25
  • $\begingroup$ Boosting is an Ensembling Training Method which creates multiple trees. See official paper, section 2.1 - arxiv.org/pdf/1603.02754.pdf $\endgroup$ – Predicted Life Dec 21 '20 at 22:30
  • $\begingroup$ The first split won't necessarily be the on the feature of highest importance, depending on your definition of importance. $\endgroup$ – Ben Reiniger Dec 21 '20 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.