RandomForest and tree feature importance in scikit-learn

Question

What is the difference between model.feature_importances_ and tree.feature_importances_ in the following code:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

# Boston Housing dataset
from sklearn.datasets import load_boston
boston = load_boston()
# Convert 'skleran.bunch' to Pandas dataframe
data = pd.DataFrame(boston.data, columns=boston.feature_names)

# Create train and test sets for cross-validation
X,y = data.iloc[:,:-1], data.iloc[:,-1]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state = 123)

model = RandomForestRegressor()
model.fit(X_train, y_train)

As I understand the following is importance of features:

importance = model.feature_importances_
importance_df = pd.DataFrame(importance, index=X_train.columns, 
                      columns=["Importance"])


        Importance
-----------------
CRIM    0.025993
ZN      0.002781
INDUS   0.004832
CHAS    0.000315
NOX     0.028655
RM      0.406285
AGE     0.017987
DIS     0.040696
RAD     0.003615
TAX     0.009281
PTRATIO 0.009103
B       0.012354
LSTAT   0.438106

And what is tree.feature_importance?:

[tree.feature_importances_ for tree in model.estimators_]

How do they differ, calculated and which is more important 0.2 or 0.9? Can't find in the docs.

Answer 1

A random forest model is an agglomeration of Decision Trees. tree.feature_importance_ defines the feature importance for each individual tree, but model.feature_importance_ is the feature importance for the forest as a whole. The docs give the explanation for calculation as:

The relative rank (i.e. depth) of a feature used as a decision node in a tree can be used to assess the relative importance of that feature with respect to the predictability of the target variable. Features used at the top of the tree contribute to the final prediction decision of a larger fraction of the input samples. The expected fraction of the samples they contribute to can thus be used as an estimate of the relative importance of the features. In scikit-learn, the fraction of samples a feature contributes to is combined with the decrease in impurity from splitting them to create a normalized estimate of the predictive power of that feature.

By averaging the estimates of predictive ability over several randomized trees one can reduce the variance of such an estimate and use it for feature selection. This is known as the mean decrease in impurity, or MDI. Refer to [L2014] for more information on MDI and feature importance evaluation with Random Forests.

The higher the number, the more important the feature

Answer 2

_feature_importance of a random forest calculates the average feature importance across all trees in the forest. While tree.feature_importances_ is the feature importance for a single tree.

Since feature importance is calculated as the contribution of a feature to maximize the split criterion (or equivalently: minimize impurity of child nodes) higher is better.

---

You can see how it works in the source code:

The property _feature_importance of random forests is defined as following (see here for the complete source code):

@property
    def feature_importances_(self):
        """
        Return the feature importances (the higher, the more important the
           feature).
        Returns
        -------
        feature_importances_ : array, shape = [n_features]
            The values of this array sum to 1, unless all trees are single node
            trees consisting of only the root node, in which case it will be an
            array of zeros.
        """
        check_is_fitted(self)

        all_importances = Parallel(n_jobs=self.n_jobs,
                                   **_joblib_parallel_args(prefer='threads'))(
            delayed(getattr)(tree, 'feature_importances_')
            for tree in self.estimators_ if tree.tree_.node_count > 1)

        if not all_importances:
            return np.zeros(self.n_features_, dtype=np.float64)

        all_importances = np.mean(all_importances,
                                  axis=0, dtype=np.float64)
        return all_importances / np.sum(all_importances)

As you can see it returns the average feature importance across all tress in the forest and thereby uses the tree class. The tree class implements this as following:

def feature_importances_(self):
        """Return the feature importances.
        The importance of a feature is computed as the (normalized) total
        reduction of the criterion brought by that feature.
        It is also known as the Gini importance.
        Returns
        -------
        feature_importances_ : ndarray of shape (n_features,)
            Normalized total reduction of criteria by feature
            (Gini importance).
        """
        check_is_fitted(self)

        return self.tree_.compute_feature_importances()

And here compute_feature_importances is defined:

cpdef compute_feature_importances(self, normalize=True):
        """Computes the importance of each feature (aka variable)."""
        cdef Node* left
        cdef Node* right
        cdef Node* nodes = self.nodes
        cdef Node* node = nodes
        cdef Node* end_node = node + self.node_count

        cdef double normalizer = 0.

        cdef np.ndarray[np.float64_t, ndim=1] importances
        importances = np.zeros((self.n_features,))
        cdef DOUBLE_t* importance_data = <DOUBLE_t*>importances.data

        with nogil:
            while node != end_node:
                if node.left_child != _TREE_LEAF:
                    # ... and node.right_child != _TREE_LEAF:
                    left = &nodes[node.left_child]
                    right = &nodes[node.right_child]

                    importance_data[node.feature] += (
                        node.weighted_n_node_samples * node.impurity -
                        left.weighted_n_node_samples * left.impurity -
                        right.weighted_n_node_samples * right.impurity)
                node += 1

        importances /= nodes[0].weighted_n_node_samples

        if normalize:
            normalizer = np.sum(importances)

            if normalizer > 0.0:
                # Avoid dividing by zero (e.g., when root is pure)
                importances /= normalizer

        return importances