I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making on new, tabular data. WLOG, the explanations generated by both methods do not agree with user intuition.

For example, when leveraging the methods in a healthcare setting, they might list the presence of a comorbidity (a disease that frequently co-occurs with the outcome disease of interest) as a factor that decreases a patient's risk of an adverse event.

Intuitively, such behavior is incorrect. We shouldn't see that history of heart attacks lowers the risk of adverse events, for example. What are some reasons that we might see these inconsistencies?

Some of my ideas

  • Class label imbalance: tried balancing the dataset, did not solve the issue
  • Kernel width for LIME: working on tuning this, but cursorily, no benefit
  • Relationship to training data: for tabular data, both lime and shap require the training dataset as input to build the explainer class. If there are instances in which a feature such as history of heart attacks were associated with a no adverse event outcome, such instances would "confuse" the methods, so to speak. However, I'm not sure I have the intuition correct there.
  • Error in understanding on my part: there may be nuances in intuition here that I've missed. Specifically, I am trying to make sure I correctly understand the relationship between the generated explanations and the training dataset used to build them.
  • $\begingroup$ do they agree with intuition when you look at the effects in aggregation? $\endgroup$
    – oW_
    Commented Jan 20, 2020 at 23:33
  • $\begingroup$ Hi @oW_, that's a good question. Unfortunately, the SHAP behavior continues to be inconsistent at an aggregate level (not sure if/how I can compute LIME at an aggregate level). For example, in a test set of 100 instances, renal disease being present is (incorrectly) showing a negative SHAP value (model impact), while in the same set, cancer being present is (correctly) showing a positive SHAP value. In this example, the positive class is a worse health outcome - hence my confusion. $\endgroup$ Commented Jan 21, 2020 at 0:50
  • 4
    $\begingroup$ Could this be a case of confounding variables? for example, if some other feature X is strongly correlated with having a disease Y, then having some other disease Z might be negatively correlated withY by itself. The intuition would be that somehow having Y and Z is rare, so conditional on Y, both X and Z are not likely, so end up having a negative relationship. $\endgroup$
    – Sean Owen
    Commented Jan 21, 2020 at 1:05
  • $\begingroup$ @SeanOwen thanks for the suggestion, will think on it. A potential non-independence of features vis a vis lime and shap is something else you've got me thinking about now too. $\endgroup$ Commented Jan 21, 2020 at 2:40

2 Answers 2


As pointed by Sean Owen, this is probably linked to correlation in the input variables.

Concerning LIME : you can read here : https://christophm.github.io/interpretable-ml-book/lime.html (paragraph 5.7) why a correlation between inputs could pose problem, more specifically due to an implementation problem. (gaussian sampling ignore features correlation).

Shapley values are designed to take correlation into account by construction (more specifically trough the symetry condition, hinting that the more variables are similar the more they will have a similar effect). However, as stated in https://arxiv.org/abs/1705.07874 (paragraph 4.2), SHAP rely on the independence of input to approximate Shapley values.

So both techniques you used are subject to caution when used on data with important correlations.


This is another version of Simpson's paradox. Frequency data should not be given undue causal interpretations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.