Evaluating machine learning explainers?

Question

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of those 'black boxes'. The explainers assign each feature an importance value for each prediction.

The thing is, those feature importance values differ from explainer to explainer, and it is rather tricky to evaluate which explainer actually performs best and produces the most sensible outcome. The authors of SHAP have tested the explainers' consistency with human intuition, assuming that:

[...] good model explanations should be consistent with explanations from humans who understand that model.

Right, that makes sense. But how to evaluate the performances of different explainers in a more generic and scalable way, without the need for human intervention?

Some authors have recently attempted to come up with evaluation metrics. I found one approach to be interesting. The authors worked with the following intuition:

[...] a feature-based local explanation has high quality only if it is difficult to change the prediction when the explanation features are kept intact. The quality of an explanation is quantified by the minimum amount of perturbation on features that are not in the explanation region in order to alter the prediction.

That makes sense, too. So I am planning to build up on that and focus on tabular data. The idea would be to, for each instance (row) in our test data:

1. Run the neural network and observe the predicted outcome.
2. Run the explainer and identify each feature's importance value.
3. Keep the features with the highest importance value intact and alter the remaining features.
4. Run the neural network again. If the predicted outcome hasn't changed, go back to step 3 and alter the features some more until a switch in the outcome.

Then, the explainer for which the necessary amount of alteration to obtain a change of outcome was the highest (aggregated across all instances), would be the best.

I was hoping to open a debate about quantitative techniques that can evaluate explainers. What do you think? Also, speaking of 'altering' the features, what would be the appropriate way to do so (when working with tabular data)? I'm open for comments and thoughts!

Answer 1

I would just like to add Microsoft's interpret to the list of explainers which uses boosting, and has some really cool graphics for explaining a model's behavior.