1
$\begingroup$

I have a dataset where I am focusing on binary classification problem. In total,I have around 60 features in my dataset

When I used Xgboost Feature Importance, I was able to see that the top 5 features account for 42% whereas rest of the of the 50 features account for 40-49 % (each feature about 1%) and remaing 8-10 features have zero importance or less than 1% of importance.

This is my best paramter list for Xgboost after gridsearch

op_params = {'alpha': [10],
 'as_pandas': [True],
 'colsample_bytree': [0.5],
 'early_stopping_rounds': [100],
 'learning_rate': [0.04],
 'max_depth': [6],
 'metrics': ['auc'],
 'num_boost_round': [10000],
 'objective': ['reg:logistic'],
 'scale_pos_weight': [3.08],
 'seed': [123],
 'subsample': [0.75]}

Since I have many low importance features, should I try to use them all in my model to increase the model metrics?

When I built the model only with top 5 features, I was able to get 80% accuracy.

I am trying to understand is it even useful to make use of these low importance feature for prediction?

Shown below is my feature importance in descending order

enter image description here

Do they even really help?

Any insights would really be helpful

$\endgroup$
1
$\begingroup$

Its all about a tradeoff.

The more you add unimportant features, the marginal will the benefits get but you risk injecting more complexity and potentially overfitting.

Ocams Razor

Also be carefull with the default feature importance approach. Read this.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Appreciate your input. Upvoted $\endgroup$ – The Great Dec 17 '19 at 12:37
  • $\begingroup$ Hi, I have a question regarding regarding Permutation importance. $\endgroup$ – The Great Dec 17 '19 at 12:47
  • $\begingroup$ Can you please put it in another question and I will gladly answer it. Also if you are satisfied with answers (generally, not neccesarry mine) dont forget to accept them. $\endgroup$ – Noah Weber Dec 17 '19 at 12:49
  • $\begingroup$ Hi, in addition does it make sense to add a 'Random number column' and any feature importance below this newly generated random number column can be ignored and rest all can be taken for model prediction. Am I correct in thinking this way? $\endgroup$ – The Great Dec 17 '19 at 13:12
  • $\begingroup$ I would not do it. Its overkill you could just use scikit-learn.org/stable/modules/generated/… with PermutationImporance packed together $\endgroup$ – Noah Weber Dec 17 '19 at 13:31
2
$\begingroup$

Adding low-value features might not help you surpass your current accuracy. Getting good quality data and add more data to the dataset or training for more epochs if it doesn't converge might help you get more accuracy.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Since there are lot of low important feature, you think it could be data quality issue? Upvoted for the help. $\endgroup$ – The Great Dec 17 '19 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.