# Xgboost : A variable specific Feature importance

I have a data set something like this:

data = [['Alex',10,13,1,0],['Bob',11,14,12,0],['Clarke',13,15,13,1],['bob',12,15,1,1]]
df = pd.DataFrame(data,columns = ["dealer","x","y","z","loss"])


I am trying to predict binary column loss, I have done this xgboost model. I got Overall feature importance. Now I need top 5 most important features dealer wise.

How to do that?

I have tried to use lime package but it is only working for Random forest.

If I get Feature importance for each observation(row) then also I can compute the feature importance dealer wise.

kindly help

Desired Output :

• Are you looking for which of the dealer categories is most predictive of a loss=1 over the entire dataset? In the example above dealer is text which makes it categorical and you handled that somehow which is not explained above. Jan 7, 2021 at 11:23
• @Craig I have edited the question. I am looking for Dealer-wise most important variables which is helping me predict loss. Jan 7, 2021 at 11:44
• You should create 3 datasets sliced on Dealer. Then get the FI for each feature. This seems the only meaningful approach. Jan 7, 2021 at 15:39
• cross-posted at stats.stackexchange.com/q/503861/232706 Jan 7, 2021 at 16:08
• @10xAI You mean to say i need to build multiple models ? as I have really less data I am not able to do that. Jan 8, 2021 at 5:42

If you use a per-observation explanation, you could just average (or aggregate in some other way) the importances of features across the samples for each Dealer.

For example, using shap to generate the per-observation explanation:

import pandas as pd
from sklearn.linear_model import LogisticRegression
import shap

data = [['Alex',10,13,1,0],['Bob',11,14,12,0],['Clarke',13,15,13,1],['Bob',12,15,1,1]]
df = pd.DataFrame(data, columns=["dealer","x","y","z","loss"])

lr = LogisticRegression()
lr.fit(df[['x', 'y', 'z']], df['loss'])

# Whatever explainer you prefer:
explainer = shap.explainers.Permutation(lr.predict_proba, df[['x', 'y', 'z']])
shap_values = explainer(df[['x', 'y', 'z']])

# get just the explanations for the positive class
shap_values = shap_values[...,1]

shap_df = pd.DataFrame(abs(shap_values.values))
shap_df.columns = ['x_shap', 'y_shap', 'z_shap']
shap_df['dealer'] = df['dealer']
shap_df.groupby('dealer').mean()


produces

dealer x_shap y_shap z_shap
Alex 0.260427 0.140054 0.075176
Bob 0.106593 0.069035 0.091146
Clarke 0.268328 0.083706 0.085807
• Hey, do you have any example of shap per observation explanation as I saw that first but i couldn't find any example on that. Jan 8, 2021 at 5:44
• Thanks This helped a lot :) Jan 11, 2021 at 12:22

What you are looking for is - "When Dealer is X, how important is each Feature."

You can try Permutation Importance.

• Can be used on fitted model
• It is Model agnostic
• Can be done for Test data too.

Slice X, Y in parts based on Dealer and get the Importance separately.

Shown for California Housing Data on Ocean_Proximity feature

from sklearn.inspection import permutation_importance

for val in x_train.ocean_proximity.unique():        # Loop on column value
x = x_train.loc[x_train.ocean_proximity==val,:] # Slices X
y = y_train.loc[x_train.ocean_proximity==val]   # Slices Y

result = permutation_importance(model, x, y, n_repeats=5, random_state=0)
result = pd.DataFrame(result.importances_mean, index= cols)

# print, sorted top 4 Features
print('Ocean--',val,result.sort_values(by=0,ascending=False)[:4])


Note - The importance value for each feature with this test and "Impurity decreased" approach are not comparable.