How is the 'feature_importance_' value calculated in sklearn random forest regressor?

Question

I have 9000 sample, with five features, and one output variable (all are numerical, continuous values). I used random forest regression method using scikit modules. I got a graph of the feature importance (using the function feature_importances_) values for each of the five features, and their sum is equal to one. I want to understand what these are, and how they are calculated mathematically. Can someone please explain?

Answer 1

Decision trees usually base feature importance on the impurity reduction achieved by splitting on the features. In classification a usual choice is gini impurity, while regression trees typically use the mean squared error or node sample variance. This is also the case in scikit learn.

For a given (binary) node $m$ with left and right child nodes the impurity reduction $Gain_{m}$ is calculated as

$$ Gain_{m} = impurity_{m} - (weight_{left} \cdot impurity_{left} + weight_{right} \cdot impurity_{right})$$

with the weights being defined as the share of the parents examples in a child node (e.g. $weight_{left} = N_{left} / N_{m}$ where $N$ is the number of examples in a node or leaf).

Now, to derive the total impurity reduction of a given feature $f$ in tree $t$ you need to sum across all nodes $m \in M_f^{(t)} $ which perform a split on that feature $f$ and divide it by the total impurity reduction number of all nodes of that tree:

$$Importance_f^{(t)} = \frac{\sum_{m \in M_f^{(t)}} Gain_m}{\sum_f\sum_{m \in M_f^{(t)}} Gain_m}$$

(Note that due to this normalization step your feature importances sum up to $1$)

Eventually, the total importance of a feature $f$ is calculated across all trees $t$ in your random forest with a total number of trees $T$: $$Importance_f = \frac{1}{T} \sum_{t=1}^TImportance_f^{(t)}$$