# How to Use Shap Kernal Explainer with Pipeline models?

I have a pandas DataFrame X. I would like to find the prediction explanation of a a particular model.

My model is given below:

pipeline = Pipeline(steps= [
('imputer', imputer_function()),
('classifier', RandomForestClassifier()
])
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)
y_pred = pipeline.fit(x_train, y_train).predict(x_test)


Now for prediction explainer, I use Kernal Explainer from Shap.

This is the following:

# use Kernel SHAP to explain test set predictions
shap.initjs()

shap_values = explainer.shap_values(x_test, nsamples=10)

# # plot the SHAP values for the Setosa output of the first instance


When I run the code, I get the error:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames.

Provided model function fails when applied to the provided data set.

ValueError: Specifying the columns using strings is only supported for pandas DataFrames


Can anyone please help me? I'm really stuck with this. Both x_train and x_test are pandas data frames.

• Your given code doesn't match the code in the error traceback; in the traceback line 4, you're initializing the KernelExplainer with np.array(x_train). May 23, 2019 at 15:33
• Ohh sorry I've edited it. @Ben Reiniger May 23, 2019 at 16:19

The reason is kernel shap sends data as numpy array which has no column names. so we need to fix it as follows:

def model_predict(data_asarray):
data_asframe =  pd.DataFrame(data_asarray, columns=feature_names)
return estimator.predict(data_asframe)


Then,

shap_kernel_explainer = shap.KernelExplainer(model_predict, x_train, link='logit')
shap_values_single = shap_kernel_explainer.shap_values(x_test.iloc[0,:])

• shap_values_single = shap_kernel_explainer.shap_values(x_test.iloc[0,:]) fails due to ValueError: Input contains NaN, infinity or a value too large for dtype('float64'). I believe this is because the test set is not being preprocessed in your code sample. Do you know how to fix this issue? Mar 1 at 15:47
• this didnt work for me until i changed estimator.predict to estimator.predict_proba Mar 1 at 19:09

I've tried to create a function as suggested but it doesn't work for my code. However, as suggested from an example on Kaggle, I found the below solution:

import shap

#load JS vis in the notebook
shap.initjs()

#set the tree explainer as the model of the pipeline
explainer = shap.TreeExplainer(pipeline['classifier'])

#apply the preprocessing to x_test
observations = pipeline['imputer'].transform(x_test)

#get Shap values from preprocessed data
shap_values = explainer.shap_values(observations)

#plot the feature importance
shap.summary_plot(shap_values, x_test, plot_type="bar")