# How to find and use the top features for XGBoost?

Suppose I have data with X_train, X_test, y_train, y_test given. As it is a classification problem I want to use XGBoost.

The issue is that there are more than 300 features.

I have found online that there are ways to find features which are important. But as I have lot of features it's causing an issue.

My current code is below. How can I modify it to say select top n ( n = 20) features and use them for training the model. I tried sorting the features based on importance but it doesn't work.

   import xgboost as xgb
gbm = xgb.XGBClassifier(max_depth=3, n_estimators=300, learning_rate=0.05).fit(X_train,y_train)
predictions = gbm.predict(X_test)

• why selecting the important features doesn't work? – Dirk Nachbar Jan 19 '18 at 13:46

I think this is what you are looking for.

results=pd.DataFrame()
results['columns']=x_data.columns
results['importances'] = clf.feature_importances_
results.sort_values(by='importances',ascending=False,inplace=True)

results[:20]


SKLearn is friendly on this. Simply with:

from sklearn.feature_selection import SelectFromModel
selection = SelectFromModel(gbm, threshold=0.03, prefit=True)
selected_dataset = selection.transform(X_test)


you will get a dataset with only the features of which the importance pass the threshold, as Numpy array. Point that the threshold is relative to the total importance, so it goes from 0 to 1.

If you want to visualize the importance, maybe to manually select the features you want, you can do like this:

xgb.plot_importance(booster=gbm ); plt.show()

• Is there a way to chose the best threshold. – John Constantine Jan 19 '18 at 19:26
• Unfortunately there is no automatic way. Cutting off features helps to regularize a model, avoiding over fitting, but too much cut make a bad model. So it depends on your data and on your model, so the only way of selecting a good threshold is with trials and error – Vincenzo Lavorini Jan 21 '18 at 8:46
• @VincenzoLavorini - So even while we use classifiers like gbm as shown above or random forest etc for feature selection, do we need to select the best hyperparamters? – The Great Dec 13 '19 at 13:08
• Or its only during model building and for feature selection it's okay to have just an estimator with default values? Upvoted as your response somehwat helped – The Great Dec 13 '19 at 13:09