Given a binary classifier, is it always possible to explain why it has classified some input as a positive class ? And by that I mean, if we have a big set of features, is there a tool that says : 'For this output, these are the features that were the most responsible for labeling it as a positive' ?

Thanks !

  • $\begingroup$ You could borrow a page from the LIME paper and locally fit a model that is interpretable; logistic regression. $\endgroup$ – Emre Jul 31 '17 at 15:33
  • $\begingroup$ A decision tree is very presentable/ interpretable. $\endgroup$ – Hobbes Aug 3 '17 at 22:32

Try out some of the examples in this library, which attempts to use machine learning to understand black box machine learning models in Python.

  • $\begingroup$ Heard about this in a talk at a conf. yesterday :D . However, trying to make non-tree algos. explain themselves is unnecessarily complex. But, that's a personal opinion though! $\endgroup$ – Dawny33 Jul 31 '17 at 11:53
  • $\begingroup$ That's what I was looking for except I'm using factorization machines for the moment being and that's not included in ELI5 unfortunately. I'll see how it performs for included algorithms nonetheless, thanks ! $\endgroup$ – mlx Jul 31 '17 at 12:06
  • $\begingroup$ @CalZ thanks for the link, its impressive will try it out. $\endgroup$ – Xformer Jul 31 '17 at 12:12

It's hard to give a very good answer to such a broad question.

Model interpretability is a big topic, and usually depends on the model. Simpler models, such as logistic regressions, are easier to interpret than neural networks. It's easy to say things like "if I can increase feature $X_i$ by 1, then the odds of $y$ happening will increase by $\beta_i$".

Likewise, individual decision trees are easier to interpret than random forests. Yet, some people try to interpret random forests by computing "feature importance", which can be computed several ways, one of which is the number of splits that include the feature relative to the number of samples it splits.

You want a way to treat your model as a black box and be able to interpret any model?

I can think of two ways:

  1. Manipulate your inputs and see what happens to the ouput, using your sample

One typical way is to change the input and see the impact on the model performance. In images, you can black out parts of the image, and see which parts contributes most to the accuracy. This is widely used for convolutional neural networks, which are hard to interpret otherwise.

For numerical variables, you can zero out or add some noise to each feature and see what the impact of that individual feature has on the result.

I have seen these kind of things widely used.

  1. Train with and without the feature

Similar to the previous one, except you train with and without the feature, and see the impact it has on the model accuracy.

This has the added benefit that you don't have to think about hyperparameters such as how much noise you'll add to each feature like the previous approach. Furthermore, you can better understand the impact of several features in the output by trying with and without those.

I haven't seen this approach being used, but apparently it's also being used, as another person replied.

Anyway, I would avoid such hacks. Most models can be made somehow interpretable. I have seen people even making recurrent neural networks interpretable. And, if interpretability is a concern, just use a simpler model that is easier to interpret and explain.


Interesting, I haven't heard of such thing yet but correlation matrix between features and the target variable will give you idea to certain extend. If correlation is on higher side then most likely those feature will have higher say while predicting the target. If you are using python I would suggest you to visualize correlation matrix using seaborn here is sample code 1.

  • $\begingroup$ This is one way to start. But then again, feature selection methods usually find out inputs that are important for generalization since it looks at the entire dataset and provides what can be a useful ensemble view. What would be cool is providing an answer to that question on an instance-basis ! $\endgroup$ – mlx Jul 31 '17 at 12:04
  • $\begingroup$ Agree, this could be very helpful to create more of robust features. This is one thread I will follow regularly for sure. :) $\endgroup$ – Xformer Jul 31 '17 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.