I've been reading that feature importance in xgboost is computed the same way as in random forests. However, the learning rate reduces the effect of downstream trees. Is the learning rate taken into consideration for the purposes of deriving feature importance?
3
-
$\begingroup$ There are multiple importance measures available in xgboost; which are you interested in? $\endgroup$– Ben Reiniger ♦Jan 17 at 22:04
-
$\begingroup$ @BenReiniger based on gini, for example, thank you. $\endgroup$– BobJan 18 at 10:02
-
$\begingroup$ I don't think gini is applicable in GBMs. xgboost offers gain, cover, and weight. $\endgroup$– Ben Reiniger ♦Jan 21 at 17:17
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Yes, the learning rate is taken into consideration for deriving feature importance. The learning rate in Xgboost is basically a weighting factor that is used when new trees are added to the model. What this means is that changing the learning rate results in an overall different structure of the model.
Feature importance in Xgboost, as you have mentioned, is calculated similar as in random forests. This is relatively straightforward, as features are simply ranked by their occurrence in the trees. Features that are used more often to make decisions have a higher feature importance.
Now lets go back to the learning rate. The whole structure of the model depends on the learning rate. This means that changing the learning rate will result in a change in how often the features are used. Therefore, a change in the learning rate results in a change in the feature importances.
This is something you can test. Create a Xgboost model, print the feature importance, change the learning rate and print it again. You will see how it changes.
$\begingroup$
$\endgroup$
No, the learning rate is not taken into consideration when deriving feature importance in XGBoost. The feature importance is calculated based on the number of times a feature is used to split the data across all trees, regardless of the learning rate. The learning rate in XGBoost is used to control the contribution of each new tree added to the model, but it does not affect the calculation of feature importance.
