# How to determine feature importance while using xgboost in pipeline?

How to determine feature importance while using xgboost (XGBclassifier or XGBregressor) in pipeline?

AttributeError: 'Pipeline' object has no attribute 'get_fscore'


The answer provided here is similar but I couldn't get the idea.

As I found, there are two ways to determine feature importance: First:

print(grid_search.best_estimator_.named_steps["clf"].feature_importances_)


result:

[ 0.14582562  0.08367272  0.06409663  0.07631433  0.08705109  0.03827286
0.0592836   0.05025916  0.07076083  0.0699278   0.04993521  0.07756387
0.05095335  0.07608293]


Second:

print(grid_search.best_estimator_.named_steps["clf"].booster().get_fscore())


result:

{'f2': 1385, 'f11': 1676, 'f12': 1101, 'f6': 1281, 'f9': 1511, 'f7': 1086, 'f5': 827, 'f0': 3151, 'f10': 1079, 'f1': 1808, 'f3': 1649, 'f13': 1644, 'f8': 1529, 'f4': 1881}


Third:

print(grid_search.best_estimator_.named_steps["clf"].get_booster().get_fscore())

• the problem I found is that the resultant feature names were not present in the training data itself. I had obviously renamed the f2 to the X2. Did you face such an issue ? Jun 24 '17 at 8:10
• @fixxxer First, I am so sorry for being late. Then, in the above example, my features don't have names so the result is reported by f1, f2, and etc. However, If I use some specific names there is no problem. For example, I do this: ln=X.shape. names = ["x%s" % i for i in range(1,ln[1]+1)]. print(sorted(zip(map(lambda x: round(x, 4), grid_search.best_estimator_.named_steps["clf"].feature_importances_),names), reverse=True)). Jun 28 '17 at 7:55
• Thank @ebrahmi. It's an issue with the Sklearn wrapper that the feature names are not present through an API. So, what worked for me is that I had to do this to get the exact column names - dict(zip(t.feature_names, train.columns)) Jun 28 '17 at 12:00
• @fixxxer Could you please let me know how to print sorted features in conjunction with their names. Next, select the five features with the most feature importance. For example, f1:20, f5:17, f7:14, f10:13,f6:10, f3:8, ... and then, f1, f5, f7, f10, f6 Dec 14 '17 at 18:41

# Getting a reference to the xgboost object

You should first get the XGBClassifier or XGBRegressor element from the pipeline. You could do this either by getting the n-th element or by specifying the name.

clf = XGBClassifier()
pipe = Pipeline([('other', other_element), ('xgboost', clf)])


To get the XGBClassifier you could either:

• use clf if you still have a reference to it
• index the pipeline by name: pipe.named_steps['xgboost']
• index the pipeline by location: pipe.steps[1]

# Getting the importance

Secondly, it seems that importance is not implemented for the sklearn implementation of xgboost. See this github issue. A solution to add this to your XGBClassifier or XGBRegressor is also offered over their. It boils down to adding the methods to the class yourself.

• Thanks. Does 'feature_importances_' do the same as 'get_fscore'? Here (stackoverflow.com/questions/38212649/…) 'feature_importances_' is used.@ Pieter Jan 1 '17 at 14:05
• If you look at the code you see that it uses get_fscore() internally Jan 1 '17 at 15:23