This post attempts to explain the difference between explainability and interpretability of ML models. However, the explanation is somewhat unclear. Can somebody provide specific examples of models that are explainable but not interpretable (or the over way round)?


I am following the definitions by Cynthia Rudin (and the article by Keith O’Rourke which is based on it) here:

  • Explainable ML – using a black box and explaining it afterwards.
  • Interpretable ML – using a model that is not black box.

Accordingly, a decision tree, for example, is interpretable since it inherently makes its decision explicit through the nodes/split points. And according to above definitions it is not explainable since it is not a black box model (interpretable models are not a subset of explainable models according to the definitions).

In contrast, a CNN, for example, is a black box model which implicitly encodes its decision making procedure. However, ex-post analysis is an approach to make such a model explainable. You can, for example, assess the feature map activations per layer to do so, as done in this article:

enter image description here

This analysis reveals, for example, that layer 2 gets activates by patterns such as edges and layer 3 detects more complex patterns. Obviously this ex-post analysis has a different quality than the explicitly encoded rules of a decision tree.

(Somewhat contradicting the given definitions you could say that explainable models require a larger degree of interpretation while interpretable models explain their decisions inherently - but that is only my wording and not how the authors of above articles phrase it.)

  • $\begingroup$ Nice distinction between both terms $\endgroup$
    – German C M
    Aug 7 at 10:59
  • $\begingroup$ Great answer! Thank you! Can I say that Local Interpretable Model-agnostic Explanations (LIME) is that second model (in the definition of Cynthia Rubin) that we create to explain the black-box neural networks? $\endgroup$
    – Qwerty
    Aug 7 at 13:40
  • $\begingroup$ @Qwerty yes, LIME would be another approach to explain a black box model. $\endgroup$
    – Sammy
    Aug 7 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.