SHAP values are explaining the wrong output value

Question

I was checking the local accuracy property of the SHAP values. It states that for a data point $(X,y)$, the SHAP values $(s_1,s_2,s_3,...)$ of features $(x_1,x_2,x_3,...)$ sum up to the difference of model output at $X$ i.e. $pred(X)$ and expected model output i.e. $ev$. So, the equation is, $$pred(X) = \sum s_i + ev$$ But, in my implementation I am getting the following relation $$logit(pred(X)) = \sum s_i + ev$$ It's a binary classification problem and I am using Light GBM so the model output i.e. $pred(x)$ is the raw output or the log-odds. Logit of log-odds does not make any sense. Here is the code:

## IMPORTS
import lightgbm as lgb
from sklearn.model_selection import train_test_split
import shap
import numpy as np
import scipy
## GET DATA and TRAIN a lightgbm model
X, y = shap.datasets.adult()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=7)
d_train = lgb.Dataset(X_train, label=y_train)
d_test = lgb.Dataset(X_test, label=y_test)
params = {
    "max_bin": 512,
    "learning_rate": 0.05,
    "boosting_type": "gbdt",
    "objective": "binary",
    "metric": "binary_logloss",
    "num_leaves": 10,
    "verbose": -1,
    "min_data": 100,
    "boost_from_average": True,
    "early_stopping_round": 50,
}
model = lgb.train(
    params,
    d_train,
    10000,
    valid_sets=[d_test],
)
## get SHAP values
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)
expected_value = explainer.expected_value
## get model predictions from lightgbm - this will give log-odds
model_output = model.predict(X)
## Checking local accuracy 
## For first record
print("shap_values[0].sum() + expected_value = ",shap_values[0].sum() + expected_value)
print("scipy.special.logit(model_output[0]) = ", scipy.special.logit(model_output[0]))
##For all records
difference_vector = (np.sum(shap_values, axis = 1) + expected_value - scipy.special.logit(model_output))
print("# of records where (sum of shap val and expected value) != logit(lgbm output):",np.where(difference_vector>1e-5,1,0).sum())
print("The explainations and expected_value should sum up to the model_output and not the logit of the model_ouput")

Output:

shap_values[0].sum() + expected_value =  -5.273629097885442
scipy.special.logit(model_output[0]) =  -5.273629097885442
# of records where (sum of shap val and expected value) != logit(lgbm output): 0
The explainations and expected_value should sum up to the model_output and not the logit of the model_ouput

What am I missing guys?

Answer 1

In the latest lgbm version, predict does not output raw scores, instead it outputs probabilities for binary classification task. The above code will work without using logits if model_output is computed like the following:

model_output = model.predict(X,raw_score = True)