One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can they be used with unsupervised learning?

Let's say we are doing anomaly detection with tabular data and we run some algorithm like Isolation Forest (or any other).

Is it conceptually right to use Shapley or Lime to try to give a local explanation of the results when using unsupervised learning?


I don't think so, not directly. SHAP is trying to explain each feature's effect on the prediction, but you have no label here. It might be better to ask therefore, what are you trying to explain?

In the case of an isolation forest, you can find the short path through the trees to any anomaly. That path tells you why the trees separated it, based on what features. It may not be super interpretable, but can be read off directly.

  • $\begingroup$ Why yes? With Isolation forest you can have an anomaly score. You can see why that prediciton was done, a see what was the feature that made it be an anomaly. $\endgroup$ Sep 10 '20 at 8:37
  • $\begingroup$ I am not sure if Isolation Forest trees are interpretable, there are a bagging of decision tree, which normally makes it not really interpretable $\endgroup$ Sep 10 '20 at 8:38
  • $\begingroup$ Still, this are some thoughts that I have had, but I don't have a solid answer on it $\endgroup$ Sep 10 '20 at 8:39
  • $\begingroup$ It's not a classifier. yes you can derive metrics for 'anomaly' from an isolation forest but it's not what the forest predicts. SHAP et al are trying to explain the direct prediction of the model, but there is none in this case. $\endgroup$
    – Sean Owen
    Sep 10 '20 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.