3
$\begingroup$

I have 9000 sample, with five features, and one output variable (all are numerical, continuous values). I used random forest regression method using scikit modules. I got a graph of the feature importance (using the function feature_importances_) values for each of the five features, and their sum is equal to one. I want to understand what these are, and how they are calculated mathematically. Can someone please explain?

$\endgroup$
1
  • 1
    $\begingroup$ You can open the file which that feature is saved in a look specifically at the formula it uses. My guess is that it is based on how much entropy each reduces in the final model...but the best way to know for sure is to open up the code and read it. $\endgroup$ Commented Jan 10, 2020 at 18:29

1 Answer 1

3
$\begingroup$

Decision trees usually base feature importance on the impurity reduction achieved by splitting on the features. In classification a usual choice is gini impurity, while regression trees typically use the mean squared error or node sample variance. This is also the case in scikit learn.

For a given (binary) node $m$ with left and right child nodes the impurity reduction $Gain_{m}$ is calculated as

$$ Gain_{m} = impurity_{m} - (weight_{left} \cdot impurity_{left} + weight_{right} \cdot impurity_{right})$$

with the weights being defined as the share of the parents examples in a child node (e.g. $weight_{left} = N_{left} / N_{m}$ where $N$ is the number of examples in a node or leaf).

Now, to derive the total impurity reduction of a given feature $f$ in tree $t$ you need to sum across all nodes $m \in M_f^{(t)} $ which perform a split on that feature $f$ and divide it by the total impurity reduction number of all nodes of that tree:

$$Importance_f^{(t)} = \frac{\sum_{m \in M_f^{(t)}} Gain_m}{\sum_f\sum_{m \in M_f^{(t)}} Gain_m}$$

(Note that due to this normalization step your feature importances sum up to $1$)

Eventually, the total importance of a feature $f$ is calculated across all trees $t$ in your random forest with a total number of trees $T$: $$Importance_f = \frac{1}{T} \sum_{t=1}^TImportance_f^{(t)}$$

$\endgroup$
3
  • 1
    $\begingroup$ This seems to be applicable only for classification. If so, how is the importance calculated in the case of regression? $\endgroup$ Commented Jan 20, 2020 at 12:02
  • $\begingroup$ @SuvardhanJonnalagadda See the first paragraph. You must change your measure of "impurity" for regression (e.g. use MSE). $\endgroup$
    – ado sar
    Commented Sep 19, 2022 at 16:31
  • $\begingroup$ @Jonathan So the average feature importances it is not guaranteed to sum to 1 right? $\endgroup$
    – ado sar
    Commented Aug 30, 2023 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.