Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
For a project, I used the feature_importances_ attributes from the RandomForestRegressor. Everything works well but I don't know how to explain why one feature is more important than another. I mean I know that the higher the score is the higher the importance, but I don't understand how it is calculated.
For exemple, if a variable as a score of 0.35 what does it mean?
I would appreciate if someone could explain me how it works!
Thanks!
scikit-learn's RandomForestRegressor feature importance is computed in each tree composing the forest. You can find the source code here (starting at line 1053).
What it does is, for each node in the tree where the split is made on the feature, it substracts each child node's (left and right) impurity values from the parent node impurity value. If impurity decreases a lot (meaning the feature performs an efficient split), it basically gives a high score. Of course, all that is weighted depending on how useful the test is for the result: a split between two individuals gives a high impurity decrease, but is trivial, so much easier than between two large populations.
Once the feature importance has been determined for each tree, it is summed up and normalized so that the feature_importances_ vector sums up to 1.
It might introduce some biaises, I guess that it should be the case if variables are not scaled, for instance. However it is quite easy to compute (you just have to read impurity values from the tree), so I guess this is why it is provided by default. But the method is not unequivocal, there are other methods out there that you can implement more or less manually:
... and probably a few other approaches. All those will give you other scores that may be of help.
