3
$\begingroup$

I have trained an XGBoost binary classifier and I would like to extract features importance for each observation I give to the model (I already have global features importance).

More specifically, I am looking for a way to determine, for each instance given to the model, which features have the most impact and make the input belong to one class or another. I would like to know something like the top 5 features which make the observation belong to some class and indications on how I should modify these 5 features so that the probability of belonging to this class decreases or increases.

For example, let’s say my model predicts whether a house costs more than 100,000 dollars (this is the positive class) based on its location, surface and number of bedrooms. I give it the following input: London, 400 square foots, 4 bedrooms and my model predicts a probability of 56% for the house to be in the positive class. I am looking for a Python module or a function that would show the most influential features for each observation.

$\endgroup$
3
$\begingroup$

I would suggest you're probably looking for something like one of these two packages:

https://github.com/slundberg/shap
https://github.com/marcotcr/lime

See this notebook of lime's for example, which shows how you can use it to see why a specific sample in your data resulted in the prediction from the model:

https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html

$\endgroup$
0
$\begingroup$

After training your model, use xgb_feature_importances_ to see the impact the features had on the training. Note that there are 3 types of how importance is calculated for the features (weight is the default type) :

  • weight : The number of times a feature is used to split the data across all trees.
  • cover : The number of times a feature is used to split the data across all trees weighted by the number of training data points that go through those splits.
  • gain : The average training loss reduction gained when using a feature for splitting.

Here's an example :

#Available importance_types = [‘weight’, ‘gain’, ‘cover’, ‘total_gain’, ‘total_cover’]
f = 'gain'
xgb.get_booster().get_score(importance_type= f)
$\endgroup$
  • 1
    $\begingroup$ I'm not sure this answers OP's question, as they state they already have global feature importance. $\endgroup$ – bradS Aug 2 '19 at 13:35
  • $\begingroup$ didn't see that. oops. $\endgroup$ – Blenz Aug 2 '19 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.