I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of those 'black boxes'. The explainers assign each feature an importance value for each prediction.

The thing is, those feature importance values differ from explainer to explainer, and it is rather tricky to evaluate which explainer actually performs best and produces the most sensible outcome. The authors of SHAP have tested the explainers' consistency with human intuition, assuming that:

[...] good model explanations should be consistent with explanations from humans who understand that model.

Right, that makes sense. But how to evaluate the performances of different explainers in a more generic and scalable way, without the need for human intervention?

Some authors have recently attempted to come up with evaluation metrics. I found one approach to be interesting. The authors worked with the following intuition:

[...] a feature-based local explanation has high quality only if it is difficult to change the prediction when the explanation features are kept intact. The quality of an explanation is quantified by the minimum amount of perturbation on features that are not in the explanation region in order to alter the prediction.

That makes sense, too. So I am planning to build up on that and focus on tabular data. The idea would be to, for each instance (row) in our test data:

  1. Run the neural network and observe the predicted outcome.
  2. Run the explainer and identify each feature's importance value.
  3. Keep the features with the highest importance value intact and alter the remaining features.
  4. Run the neural network again. If the predicted outcome hasn't changed, go back to step 3 and alter the features some more until a switch in the outcome.

Then, the explainer for which the necessary amount of alteration to obtain a change of outcome was the highest (aggregated across all instances), would be the best.

I was hoping to open a debate about quantitative techniques that can evaluate explainers. What do you think? Also, speaking of 'altering' the features, what would be the appropriate way to do so (when working with tabular data)? I'm open for comments and thoughts!

  • $\begingroup$ This is a great project, and one I'd like to hear more about, but it doesn't seem suited to the stackexchange model. $\endgroup$
    – Ben Reiniger
    Dec 20, 2019 at 22:23
  • $\begingroup$ content is huge now. Almost 50 algos are available- machinelearningstories.blogspot.com $\endgroup$ Jun 3, 2020 at 9:04

1 Answer 1


I would just like to add Microsoft's interpret to the list of explainers which uses boosting, and has some really cool graphics for explaining a model's behavior.

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.