In a classification problem, suppose a model has 2 variables, A and B, and the null model (the model without any variable) predicts 50% probability for belonging to class 1 for all the instances.

Now suppose that for all the data instances xi, feature A makes the model to predict the opposite value of the correct class with a large magnitude(i.e. if the true class is 1 and the null model predicts 0.5, A makes the model to predict 0.1 probability), and B makes the model to predict toward the correct class but with less magnitude (i.e. if the true class is 1 and the null model predicts 0.5, adding B makes the model to predict 0.57). This means that the feature **B is actually a better feature than A but the impact of A is stronger than B.

One common way of calculating the feature importance is calculating the average of the absolute Shapley values for all the instances for feature A and B. The example that I just gave shows that this method of finding feature importance using mean absolute value would give wrong results since it would give feature A a higher importance that feature B.

Am I wrong?


1 Answer 1


I think you are correct in your analysis, but have used a very unusual context. It would be an impressively bad model that uses $A$ confidently in the wrong direction.

If such a situation did occur, then shap is still answering the question "which features are impactful on the output?" correctly: $A$ has the larger impact on decisions. Shap can only tell you about the model, and that only proxies for reality if the model is reasonably good.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.