4
$\begingroup$

Suppose I have data with X_train, X_test, y_train, y_test given. As it is a classification problem I want to use XGBoost.

The issue is that there are more than 300 features.

I have found online that there are ways to find features which are important. But as I have lot of features it's causing an issue.

My current code is below. How can I modify it to say select top n ( n = 20) features and use them for training the model. I tried sorting the features based on importance but it doesn't work. 

   import xgboost as xgb
    gbm = xgb.XGBClassifier(max_depth=3, n_estimators=300, learning_rate=0.05).fit(X_train,y_train)
    predictions = gbm.predict(X_test)
Improve this question
$\endgroup$
1
  • $\begingroup$ why selecting the important features doesn't work? $\endgroup$
    – Dirk N
    Jan 19, 2018 at 13:46

2 Answers 2

Reset to default
5
$\begingroup$

SKLearn is friendly on this. Simply with:

from sklearn.feature_selection import SelectFromModel
selection = SelectFromModel(gbm, threshold=0.03, prefit=True)
selected_dataset = selection.transform(X_test)

you will get a dataset with only the features of which the importance pass the threshold, as Numpy array. Point that the threshold is relative to the total importance, so it goes from 0 to 1.

If you want to visualize the importance, maybe to manually select the features you want, you can do like this:

xgb.plot_importance(booster=gbm ); plt.show()
Improve this answer
$\endgroup$
4
  • $\begingroup$ Is there a way to chose the best threshold. $\endgroup$
    – John Constantine
    Jan 19, 2018 at 19:26
  • 1
    $\begingroup$ Unfortunately there is no automatic way. Cutting off features helps to regularize a model, avoiding over fitting, but too much cut make a bad model. So it depends on your data and on your model, so the only way of selecting a good threshold is with trials and error $\endgroup$
    – Vincenzo Lavorini
    Jan 21, 2018 at 8:46
  • $\begingroup$ @VincenzoLavorini - So even while we use classifiers like gbm as shown above or random forest etc for feature selection, do we need to select the best hyperparamters? $\endgroup$
    – The Great
    Dec 13, 2019 at 13:08
  • $\begingroup$ Or its only during model building and for feature selection it's okay to have just an estimator with default values? Upvoted as your response somehwat helped $\endgroup$
    – The Great
    Dec 13, 2019 at 13:09
5
$\begingroup$

I think this is what you are looking for. 

results=pd.DataFrame()
results['columns']=x_data.columns
results['importances'] = clf.feature_importances_
results.sort_values(by='importances',ascending=False,inplace=True)

results[:20]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.