I'm trying to understand exactly how feature_importances in scikit-learn's RandomForestClassifier works. I managed to find this helpful link explaining most of the process: https://towardsdatascience.com/the-mathematics-of-decision-trees-random-forest-and-feature-importance-in-scikit-learn-and-spark-f2861df67e3
I just have two questions about the equation for ni_j (equation for node importance, first equation in the section on feature importance):
- What is meant by weighted number of samples reaching node j? Are they running all the training samples through the tree and counting how many reach node j? If so, why is the word 'weighted' necessary?
- Can we be mathematically certain that ni_j >= 0?
Edit: Looking at the code for forest.py, I see this rather mystifying function:
def feature_importances_(self): """Return the feature importances (the higher, the more important the feature). Returns ------- feature_importances_ : array, shape = [n_features] """ check_is_fitted(self, 'estimators_') all_importances = Parallel(n_jobs=self.n_jobs, backend="threading")( delayed(getattr)(tree, 'feature_importances_') for tree in self.estimators_) return sum(all_importances) / len(self.estimators_)
I can see it's calculating a feature importance at each tree, and then taking the mean over all trees. But how it manages to calculate the feature importance at a tree without calling any functions to count samples or compute impurities, I don't understand.