I have 9000 sample, with five features, and one output variable (all are numerical, continuous values). I used random forest regression method using scikit modules. I got a graph of the feature importance (using the function feature_importances_) values for each of the five features, and their sum is equal to one. I want to understand what these are, and how they are calculated mathematically. Can someone please explain?

  • 1
    $\begingroup$ You can open the file which that feature is saved in a look specifically at the formula it uses. My guess is that it is based on how much entropy each reduces in the final model...but the best way to know for sure is to open up the code and read it. $\endgroup$ – bethanyP Jan 10 '20 at 18:29

Decision trees usually base feature importance on the impurity reduction achieved by splitting on the features. In classification a usual choice is gini impurity, while regression trees typically use the mean squared error or node sample variance. This is also the case in scikit learn.

For a given (binary) node $m$ with left and right child nodes the impurity reduction $Gain_{m}$ is calculated as

$$ Gain_{m} = impurity_{m} - (weight_{left} \cdot impurity_{left} + weight_{right} \cdot impurity_{right})$$

with the weights being defined as the share of the parents examples in a child node (e.g. $weight_{left} = N_{left} / N_{m}$ where $N$ is the number of examples in a node or leaf).

Now, to derive the total impurity reduction of a given feature $f$ in tree $t$ you need to sum across all nodes $m \in M_f^{(t)} $ which perform a split on that feature $f$ and divide it by the total impurity reduction number of all nodes of that tree:

$$Importance_f^{(t)} = \frac{\sum_{m \in M_f^{(t)}} Gain_m}{\sum_f\sum_{m \in M_f^{(t)}} Gain_m}$$

(Note that due to this normalization step your feature importances sum up to $1$)

Eventually, the total importance of a feature $f$ is calculated across all trees $t$ in your random forest with a total number of trees $T$: $$Importance_f = \frac{1}{T} \sum_{t=1}^TImportance_f^{(t)}$$

  • $\begingroup$ This seems to be applicable only for classification. If so, how is the importance calculated in the case of regression? $\endgroup$ – Suvardhan Jonnalagadda Jan 20 '20 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.