Using a random forest is it possible to determine which features were the driving features to classify a specific sample as class A?

I know I can ask which features are more important to perform classification of ANY sample, but can I ask this for a specific sample? E.g. Why was sample 1 classified as A? Which of its features were much more like class A than class B?

Does it even make sense to ask this question of a random forest?

Bonus points on how to do it with sklearn in python :)


Question has been answered in a crosspost here: https://stats.stackexchange.com/questions/174229/feature-importance-for-random-forest-classification-of-a-sample

Python implementation here: http://blog.datadive.net/random-forest-interpretation-with-scikit-learn/

  • $\begingroup$ Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. $\endgroup$ – D.W. Jan 4 '17 at 23:09
  • $\begingroup$ This question is an edge-case that is equally appropriate for both communities without changing the wording, and from which both communities may benefit. See here, as well as other answers in the page you linked. Specifically, as the Data Science community is still small and a beta community, it can benefit from questions like this. Otherwise, it always makes more sense to post on Cross Validated due to its higher activity, and can lead to this community staying small and inactive. $\endgroup$ – CHP Jan 5 '17 at 4:56
  • $\begingroup$ I'm voting to close this question as a cross-post, where the other site's answer appears more canonical. $\endgroup$ – Sean Owen Jan 6 '17 at 17:58
  • $\begingroup$ Agree with @CHP though I agree more that we shouldn't cross-post. I think the volume of questions at this point is OK here. $\endgroup$ – Sean Owen Jan 6 '17 at 17:59
  • $\begingroup$ Indeed, the question was posted over a year ago, things have changed since. I do worry that it may discourage methodology questions as they will always be highly appropriate for both CV and DS communities. I would always favor CV as I typically get quicker and beefier answers there regarding methodology. As the meta links suggest, the overall issue is that SE was not designed for overlapping communities, and I suppose there's not a lot we can do about that. $\endgroup$ – CHP Jan 6 '17 at 19:32

So let me clarify your query: you have trained a random forest model to classify a dataset into multiple classes, e.g. A, B, C, or D. Now, you want to understand for each different class label (e.g. A), which features contributed to it being classified as class A vs. not class A, and so on.

To find the importance of a variable in random forest, each variable is permuted among all trees and the difference in out of sample error of before and after permutation is calculated. The variables with highest difference are considered most important, and ones with lower values are less important. this method gives importance for the entire multinomial classification, i.e. A vs. B/C/D, B vs. A/C/D, etc. and not just for one label. So you can't claim that feature1 is more important to identify class A, but feature 2 is more important to identify Class B.

I've been using the random forest package implemented in R and it does not seem to provide any way to decipher the internal details on the one-vs-all categorization.

If you're really interested in doing just that, then you should fit a set of random forest models each for classifying class A vs. the rest, class B vs. the rest, etc. The variable importance of these separate models would give you the feature importance for classifying each label.

You can use the same library to fit these models using scikit-learn. You may refer to their help page here - http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

| improve this answer | |
  • 2
    $\begingroup$ This is not exactly what I was asking. I was asking for each sample you feed a trained RF and ask it to predict a class or regression value, what features contribute to that specific sample being assigned that class or value . This question was partly answered on crossvalidated where I crossposted here: stats.stackexchange.com/questions/174229/… . Additional information and an implementation can be found here: blog.datadive.net/… $\endgroup$ – CHP Jun 24 '16 at 7:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.