0
$\begingroup$

I ran into this problem:

A XGBoost model(.pickle file , constrcuted under V0.7.post3) with 100 features in it ; But I found 55 features in model (model.feature_importances_) show 0 feature importance (like matrix below);

Additionally, when I transformed the pickle file to PMML(to launch online), only 45 features in PMML file (those ones with importance>0 apparently);

So, my question is:

--why features with importance equal to 0 ending up in a XGB model ? And why they remain in the model, if they don't actually contribute to/participate in split?

## Feature importance maxtrix from model for demo
array([0.06586827, 0.04191617, 0.08383234, 0.05988024, 0.07784431,
   0.04790419, 
   ...
 
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ]
$\endgroup$

2 Answers 2

0
$\begingroup$

XGBoost Feature importance gives feature importance for all the features irrespective of they contribute to the model or not. So if your train data has 100 variable xgboost feature importance will give 100 feature importances though variables which are not important will be zero

$\endgroup$
2
  • $\begingroup$ ...say I have a pool of 1000 features, then I trained a model by narrowing to 100 features; but amoung 100, 60 are with 0 importance---- so why are these 60 vars selected in the finalized model file, since they don't actually appear in splits? $\endgroup$
    – leveygao
    Jan 27 at 12:17
  • $\begingroup$ This is more on how it is implemented, i agree with what you are saying that 60 should not be used but in XGBoost or any implementation all variables are selected and needed for final prediction $\endgroup$ Jan 27 at 12:19
0
$\begingroup$

xgboost simply displays the importance of the functions of the dataset on which it was trained, no more. If some functions are so bad that they are not included in any of the trees, then their importance will be 0. You can submit a dataset of size (a x b), and specify n_estimators=1,max_depth=1 and you will see that 1 function will have an importance of 1, all the others 0, because that they didn't even have a chance to get into the trees.

You can use some feature selection algorithms to get rid of unnecessary functions, such as boruta, recursive feature elimination, etc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.