# Reasons that LIME and SHAP might not agree with intuition

I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making on new, tabular data. WLOG, the explanations generated by both methods do not agree with user intuition.

For example, when leveraging the methods in a healthcare setting, they might list the presence of a comorbidity (a disease that frequently co-occurs with the outcome disease of interest) as a factor that decreases a patient's risk of an adverse event.

Intuitively, such behavior is incorrect. We shouldn't see that history of heart attacks lowers the risk of adverse events, for example. What are some reasons that we might see these inconsistencies?

Some of my ideas

• Class label imbalance: tried balancing the dataset, did not solve the issue
• Kernel width for LIME: working on tuning this, but cursorily, no benefit
• Relationship to training data: for tabular data, both lime and shap require the training dataset as input to build the explainer class. If there are instances in which a feature such as history of heart attacks were associated with a no adverse event outcome, such instances would "confuse" the methods, so to speak. However, I'm not sure I have the intuition correct there.
• Error in understanding on my part: there may be nuances in intuition here that I've missed. Specifically, I am trying to make sure I correctly understand the relationship between the generated explanations and the training dataset used to build them.
• do they agree with intuition when you look at the effects in aggregation? – oW_ Jan 20 at 23:33
• Hi @oW_, that's a good question. Unfortunately, the SHAP behavior continues to be inconsistent at an aggregate level (not sure if/how I can compute LIME at an aggregate level). For example, in a test set of 100 instances, renal disease being present is (incorrectly) showing a negative SHAP value (model impact), while in the same set, cancer being present is (correctly) showing a positive SHAP value. In this example, the positive class is a worse health outcome - hence my confusion. – AmeySMahajan Jan 21 at 0:50
• Could this be a case of confounding variables? for example, if some other feature X is strongly correlated with having a disease Y, then having some other disease Z might be negatively correlated withY by itself. The intuition would be that somehow having Y and Z is rare, so conditional on Y, both X and Z are not likely, so end up having a negative relationship. – Sean Owen Jan 21 at 1:05
• @SeanOwen thanks for the suggestion, will think on it. A potential non-independence of features vis a vis lime and shap is something else you've got me thinking about now too. – AmeySMahajan Jan 21 at 2:40