1
$\begingroup$

Suppose I have data with X_train, X_test, y_train, y_test given. As it is a classification problem I want to use XGBoost.

The issue is that there are more than 300 features.

I have found online that there are ways to find features which are important. But as I have lot of features it's causing an issue.

My current code is below. How can I modify it to say select top n ( n = 20) features and use them for training the model. I tried sorting the features based on importance but it doesn't work.

   import xgboost as xgb
    gbm = xgb.XGBClassifier(max_depth=3, n_estimators=300, learning_rate=0.05).fit(X_train,y_train)
    predictions = gbm.predict(X_test)
$\endgroup$
  • $\begingroup$ why selecting the important features doesn't work? $\endgroup$ – Dirk Nachbar Jan 19 '18 at 13:46
2
$\begingroup$

SKLearn is friendly on this. Simply with:

from sklearn.feature_selection import SelectFromModel
selection = SelectFromModel(gbm, threshold=0.03, prefit=True)
selected_dataset = selection.transform(X_test)

you will get a dataset with only the features of which the importance pass the threshold, as Numpy array. Point that the threshold is relative to the total importance, so it goes from 0 to 1.

If you want to visualize the importance, maybe to manually select the features you want, you can do like this:

xgb.plot_importance(booster=gbm ); plt.show()
$\endgroup$
  • $\begingroup$ Is there a way to chose the best threshold. $\endgroup$ – John Constantine Jan 19 '18 at 19:26
  • 1
    $\begingroup$ Unfortunately there is no automatic way. Cutting off features helps to regularize a model, avoiding over fitting, but too much cut make a bad model. So it depends on your data and on your model, so the only way of selecting a good threshold is with trials and error $\endgroup$ – Vincenzo Lavorini Jan 21 '18 at 8:46
1
$\begingroup$

I think this is what you are looking for.

results=pd.DataFrame()
results['columns']=x_data.columns
results['importances'] = clf.feature_importances_
results.sort_values(by='importances',ascending=False,inplace=True)

results[:20]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.