Let say I got feature importance for xgclassifier

sorted(zip(xgb.feature_importances_, X.columns), reverse=True)

[(0.10650729, 'modelMag_i'),
 (0.08187373, 'psfMag_g'),
 (0.070714064, 'modelVar'),
 (0.06747197, 'modelMag_z'),
 (0.061302684, 'fiberMag_g'),
 (0.05923392, 'fibVar'),
 (0.057112347, 'psfMag_u'),
 (0.05275245, 'psfMag_r'),
 (0.047756154, 'modelMag_g'),
 (0.046770878, 'psfMag_z'),
 (0.034744404, 'modelMag_r'),
 (0.034687676, 'psfMag_i'),
 (0.032622278, 'petroMag_i'),
 (0.028391415, 'modelMag_u'),
 (0.025683628, 'petroMag_r'),
 (0.024703711, 'petroMag_z'),
 (0.022656566, 'fiberMag_z'),
 (0.021865964, 'petroMag_g'),
 (0.01854887, 'fiberMag_r'),
 (0.018389946, 'fiberMag_u'),
 (0.01721868, 'modelMean'),
 (0.016091293, 'fiberMag_i'),
 (0.013110901, 'fibMean'),
 (0.011618578, 'modelSum'),
 (0.010491995, 'fiberID'),
 (0.008898865, 'fibSum'),
 (0.008779789, 'petroMag_u')]

is removing the lowest feature will improve for xgboost or lgb classifier? or xgboost or lgb does not matter with feature importance


There is no certain answer, only trial and error. Though it should help.

Let me elaborate. Feature importance shows the impact of features on the quality of the model: the number of times there was a split using this feature or gains from splitting on this feature. The better is the feature, the higher is the importance. But some features could be important due to interactions with other features. Also if some features have a high correlation between them, the importance value could be split between them. In general removing low importance features should have a small impact on the metric. It could be positive or negative. Just try and see :)

  • 1
    $\begingroup$ Thank you. So there is no such guide line or prove 'yet' for this topic right? is there any good or recent research paper abut this topic? $\endgroup$ – slowmonk Feb 18 '20 at 4:16
  • $\begingroup$ Yes, there is no clear rule in this case. I think this article could be useful: explained.ai/rf-importance $\endgroup$ – Andrey Lukyanenko Feb 18 '20 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.