Is it valid to compare SHAP values across models?

Question

Let's say I have three models:

• a random forest with 100 trees
• a random forest with 1000 trees
• an xgboost model.

I can rank the importance of my features on my dataset for each model using SHAP, and compare relative importance across models. What is not clear is whether I can meaningfully compare the actual numerical SHAP values across models.

I am using the same features for all models.

Answer 1

Shapley values were designed in the context of game theory (source), to share value created by a coalition of player in a game. It has multiple properties, including linearity. The linearity ensure that if you were to average your models, the resulting Shapley value would be the average of Shapley values for individual models. Shapley values are comparable in that sense of considering an average model.

I think the general answer would be the opposite. Inutitively, because looking at an individual Shapley value you can't know if the value is due to an 'individual performance' or 'overall performance' of the coalition. So looking at two values in the context of ML, the difference might be both explained by a different contribution of the individual value but also by a difference in the overall performance of the model. So I would avoid doing that in the general case. (But I often do so - along with comparing individual predictions - to check if two models with similar overall performance have learned the same things or not)

Overall I would suggest you use a more suited criteria (like an information criteria) for your model selection, then use Shapley value to explain the model you selected. Not using Shapley Values to do some sort of model selection.

Note that I am mainly talking about Shapley values and not SHAP, which is an approximation. You need to be cautious with SHAP as the approximation rely on the absence of correlation between your features, which rarely happen in practice.