_feature_importance of a random forest calculates the average feature importance across all trees in the forest. While
tree.feature_importances_ is the feature importance for a single tree.
Since feature importance is calculated as the contribution of a feature to maximize the split criterion (or equivalently: minimize impurity of child nodes) higher is better.
You can see how it works in the source code:
_feature_importance of random forests is defined as following (see here for the complete source code):
Return the feature importances (the higher, the more important the
feature_importances_ : array, shape = [n_features]
The values of this array sum to 1, unless all trees are single node
trees consisting of only the root node, in which case it will be an
array of zeros.
all_importances = Parallel(n_jobs=self.n_jobs,
for tree in self.estimators_ if tree.tree_.node_count > 1)
if not all_importances:
return np.zeros(self.n_features_, dtype=np.float64)
all_importances = np.mean(all_importances,
return all_importances / np.sum(all_importances)
As you can see it returns the average feature importance across all tress in the forest and thereby uses the tree class. The tree class implements this as following:
"""Return the feature importances.
The importance of a feature is computed as the (normalized) total
reduction of the criterion brought by that feature.
It is also known as the Gini importance.
feature_importances_ : ndarray of shape (n_features,)
Normalized total reduction of criteria by feature
compute_feature_importances is defined:
cpdef compute_feature_importances(self, normalize=True):
"""Computes the importance of each feature (aka variable)."""
cdef Node* left
cdef Node* right
cdef Node* nodes = self.nodes
cdef Node* node = nodes
cdef Node* end_node = node + self.node_count
cdef double normalizer = 0.
cdef np.ndarray[np.float64_t, ndim=1] importances
importances = np.zeros((self.n_features,))
cdef DOUBLE_t* importance_data = <DOUBLE_t*>importances.data
while node != end_node:
if node.left_child != _TREE_LEAF:
# ... and node.right_child != _TREE_LEAF:
left = &nodes[node.left_child]
right = &nodes[node.right_child]
importance_data[node.feature] += (
node.weighted_n_node_samples * node.impurity -
left.weighted_n_node_samples * left.impurity -
right.weighted_n_node_samples * right.impurity)
node += 1
importances /= nodes.weighted_n_node_samples
normalizer = np.sum(importances)
if normalizer > 0.0:
# Avoid dividing by zero (e.g., when root is pure)
importances /= normalizer