Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Given a binary classifier, is it always possible to explain why it has classified some input as a positive class ? And by that I mean, if we have a big set of features, is there a tool that says : 'For this output, these are the features that were the most responsible for labeling it as a positive' ?
Thanks !
Try out some of the examples in this library, which attempts to use machine learning to understand black box machine learning models in Python:
It's hard to give a very good answer to such a broad question.
Model interpretability is a big topic, and usually depends on the model. Simpler models, such as logistic regressions, are easier to interpret than neural networks. It's easy to say things like "if I can increase feature $X_i$ by 1, then the odds of $y$ happening will increase by $\beta_i$".
Likewise, individual decision trees are easier to interpret than random forests. Yet, some people try to interpret random forests by computing "feature importance", which can be computed several ways, one of which is the number of splits that include the feature relative to the number of samples it splits.
You want a way to treat your model as a black box and be able to interpret any model?
I can think of two ways:
One typical way is to change the input and see the impact on the model performance. In images, you can black out parts of the image, and see which parts contributes most to the accuracy. This is widely used for convolutional neural networks, which are hard to interpret otherwise.
For numerical variables, you can zero out or add some noise to each feature and see what the impact of that individual feature has on the result.
I have seen these kind of things widely used.
Similar to the previous one, except you train with and without the feature, and see the impact it has on the model accuracy.
This has the added benefit that you don't have to think about hyperparameters such as how much noise you'll add to each feature like the previous approach. Furthermore, you can better understand the impact of several features in the output by trying with and without those.
I haven't seen this approach being used, but apparently it's also being used, as another person replied.
Anyway, I would avoid such hacks. Most models can be made somehow interpretable. I have seen people even making recurrent neural networks interpretable. And, if interpretability is a concern, just use a simpler model that is easier to interpret and explain.
Interesting, I haven't heard of such thing yet but correlation matrix between features and the target variable will give you idea to certain extend. If correlation is on higher side then most likely those feature will have higher say while predicting the target. If you are using python I would suggest you to visualize correlation matrix using seaborn here is sample code 1.
