5
$\begingroup$

How to determine feature importance while using xgboost (XGBclassifier or XGBregressor) in pipeline?

AttributeError: 'Pipeline' object has no attribute 'get_fscore'

The answer provided here is similar but I couldn't get the idea.

$\endgroup$
5
$\begingroup$

As I found, there are two ways to determine feature importance: First:

print(grid_search.best_estimator_.named_steps["clf"].feature_importances_)

result:

[ 0.14582562  0.08367272  0.06409663  0.07631433  0.08705109  0.03827286
  0.0592836   0.05025916  0.07076083  0.0699278   0.04993521  0.07756387
  0.05095335  0.07608293]

Second:

print(grid_search.best_estimator_.named_steps["clf"].booster().get_fscore())

result:

{'f2': 1385, 'f11': 1676, 'f12': 1101, 'f6': 1281, 'f9': 1511, 'f7': 1086, 'f5': 827, 'f0': 3151, 'f10': 1079, 'f1': 1808, 'f3': 1649, 'f13': 1644, 'f8': 1529, 'f4': 1881}

Third:

print(grid_search.best_estimator_.named_steps["clf"].get_booster().get_fscore())
| improve this answer | |
$\endgroup$
  • $\begingroup$ the problem I found is that the resultant feature names were not present in the training data itself. I had obviously renamed the f2 to the X2. Did you face such an issue ? $\endgroup$ – fixxxer Jun 24 '17 at 8:10
  • $\begingroup$ @fixxxer First, I am so sorry for being late. Then, in the above example, my features don't have names so the result is reported by f1, f2, and etc. However, If I use some specific names there is no problem. For example, I do this: ln=X.shape. names = ["x%s" % i for i in range(1,ln[1]+1)]. print(sorted(zip(map(lambda x: round(x, 4), grid_search.best_estimator_.named_steps["clf"].feature_importances_),names), reverse=True)). $\endgroup$ – ebrahimi Jun 28 '17 at 7:55
  • $\begingroup$ Thank @ebrahmi. It's an issue with the Sklearn wrapper that the feature names are not present through an API. So, what worked for me is that I had to do this to get the exact column names - dict(zip(t.feature_names, train.columns)) $\endgroup$ – fixxxer Jun 28 '17 at 12:00
  • $\begingroup$ @fixxxer Could you please let me know how to print sorted features in conjunction with their names. Next, select the five features with the most feature importance. For example, f1:20, f5:17, f7:14, f10:13,f6:10, f3:8, ... and then, f1, f5, f7, f10, f6 $\endgroup$ – ebrahimi Dec 14 '17 at 18:41
2
$\begingroup$

Getting a reference to the xgboost object

You should first get the XGBClassifier or XGBRegressor element from the pipeline. You could do this either by getting the n-th element or by specifying the name.

clf = XGBClassifier()
pipe = Pipeline([('other', other_element), ('xgboost', clf)])

To get the XGBClassifier you could either:

  • use clf if you still have a reference to it
  • index the pipeline by name: pipe.named_steps['xgboost']
  • index the pipeline by location: pipe.steps[1]

Getting the importance

Secondly, it seems that importance is not implemented for the sklearn implementation of xgboost. See this github issue. A solution to add this to your XGBClassifier or XGBRegressor is also offered over their. It boils down to adding the methods to the class yourself.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. Does 'feature_importances_' do the same as 'get_fscore'? Here (stackoverflow.com/questions/38212649/…) 'feature_importances_' is used.@ Pieter $\endgroup$ – ebrahimi Jan 1 '17 at 14:05
  • $\begingroup$ If you look at the code you see that it uses get_fscore() internally $\endgroup$ – Pieter Jan 1 '17 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.