How to determine feature importance while using xgboost (XGBclassifier or XGBregressor) in pipeline?
AttributeError: 'Pipeline' object has no attribute 'get_fscore'
The answer provided here is similar but I couldn't get the idea.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
As I found, there are two ways to determine feature importance: First:
print(grid_search.best_estimator_.named_steps["clf"].feature_importances_)
result:
[ 0.14582562 0.08367272 0.06409663 0.07631433 0.08705109 0.03827286
0.0592836 0.05025916 0.07076083 0.0699278 0.04993521 0.07756387
0.05095335 0.07608293]
Second:
print(grid_search.best_estimator_.named_steps["clf"].booster().get_fscore())
result:
{'f2': 1385, 'f11': 1676, 'f12': 1101, 'f6': 1281, 'f9': 1511, 'f7': 1086, 'f5': 827, 'f0': 3151, 'f10': 1079, 'f1': 1808, 'f3': 1649, 'f13': 1644, 'f8': 1529, 'f4': 1881}
Third:
print(grid_search.best_estimator_.named_steps["clf"].get_booster().get_fscore())
-
$\begingroup$ the problem I found is that the resultant feature names were not present in the training data itself. I had obviously renamed the f2 to the X2. Did you face such an issue ? $\endgroup$– fixxxerCommented Jun 24, 2017 at 8:10
-
$\begingroup$ @fixxxer First, I am so sorry for being late. Then, in the above example, my features don't have names so the result is reported by f1, f2, and etc. However, If I use some specific names there is no problem. For example, I do this:
ln=X.shape.names = ["x%s" % i for i in range(1,ln[1]+1)].print(sorted(zip(map(lambda x: round(x, 4), grid_search.best_estimator_.named_steps["clf"].feature_importances_),names), reverse=True)). $\endgroup$– ebrahimiCommented Jun 28, 2017 at 7:55 -
$\begingroup$ Thank @ebrahmi. It's an issue with the Sklearn wrapper that the feature names are not present through an API. So, what worked for me is that I had to do this to get the exact column names -
dict(zip(t.feature_names, train.columns))$\endgroup$– fixxxerCommented Jun 28, 2017 at 12:00 -
$\begingroup$ @fixxxer Could you please let me know how to print sorted features in conjunction with their names. Next, select the five features with the most feature importance. For example, f1:20, f5:17, f7:14, f10:13,f6:10, f3:8, ... and then, f1, f5, f7, f10, f6 $\endgroup$– ebrahimiCommented Dec 14, 2017 at 18:41
$\begingroup$
$\endgroup$
2
Getting a reference to the xgboost object
You should first get the XGBClassifier or XGBRegressor element from the pipeline. You could do this either by getting the n-th element or by specifying the name.
clf = XGBClassifier()
pipe = Pipeline([('other', other_element), ('xgboost', clf)])
To get the XGBClassifier you could either:
- use
clfif you still have a reference to it - index the pipeline by name:
pipe.named_steps['xgboost'] - index the pipeline by location:
pipe.steps[1]
Getting the importance
Secondly, it seems that importance is not implemented for the sklearn implementation of xgboost. See this github issue. A solution to add this to your XGBClassifier or XGBRegressor is also offered over their. It boils down to adding the methods to the class yourself.
-
$\begingroup$ Thanks. Does 'feature_importances_' do the same as 'get_fscore'? Here (stackoverflow.com/questions/38212649/…) 'feature_importances_' is used.@ Pieter $\endgroup$– ebrahimiCommented Jan 1, 2017 at 14:05
-
$\begingroup$ If you look at the code you see that it uses
get_fscore()internally $\endgroup$– PieterCommented Jan 1, 2017 at 15:23