scikit-learn
's RandomForestRegressor
feature importance is computed in each tree composing the forest. You can find the source code here (starting at line 1053).
What it does is, for each node in the tree where the split is made on the feature, it substracts each child node's (left and right) impurity values from the parent node impurity value. If impurity decreases a lot (meaning the feature performs an efficient split), it basically gives a high score. Of course, all that is weighted depending on how useful the test is for the result: a split between two individuals gives a high impurity decrease, but is trivial, so much easier than between two large populations.
Once the feature importance has been determined for each tree, it is summed up and normalized so that the feature_importances_
vector sums up to 1.
It might introduce some biaises, I guess that it should be the case if variables are not scaled, for instance. However it is quite easy to compute (you just have to read impurity values from the tree), so I guess this is why it is provided by default. But the method is not unequivocal, there are other methods out there that you can implement more or less manually:
- shuffling a feature's values in the dataset
- reverting the result of each test based on the feature
... and probably a few other approaches. All those will give you other scores that may be of help.