I have a dataset whose feature importance lists descendingly as 0.25, 0.16, 0.11, 0.08, 0.05, followed by many features with 0 importance, and the importance drops dramatically. Although the total importance adds up to 0.99(for obvious reason), since except for these top 5 features listed above, all others have a rather low importance fluctuating around 0.01, which is not that supportive, but removing these non-zero important features would decrease model's general performance, so keeping only the best features do not seem like a wise way.

How can we remedy this situation?

  • $\begingroup$ How are you measuring feature importance? And what model are you using? And what are the total number of features? $\endgroup$
    – Jon Nordby
    Commented Apr 30 at 22:13
  • $\begingroup$ Have you tried keeping just the top10 features? How much does performance drop? $\endgroup$
    – Jon Nordby
    Commented Apr 30 at 22:13
  • $\begingroup$ I measured feature importance using scikit learn's random forest's built-in feature importance attribute. I have about 40 features, but half of them have a less than 0.01 importance so i guess they dont contribute too much, most of the importance comes from the top 20 features, the highest importance is 0.25, and like i mentioned in the main post, the following importance drop significantly to 0.16 for the 2nd place, 0.1 for the 3rd, and 0.05~0.03 for the rest of top 20 features. The top 10 features would add up to about 0.6 importance, which is not quite convincing.. $\endgroup$
    – Yuuya
    Commented May 1 at 6:02
  • $\begingroup$ Why is it "not convincing"? Have you actually tried dropping them - how much does performance actually drop? $\endgroup$
    – Jon Nordby
    Commented May 1 at 15:11
  • $\begingroup$ RandomForest feature importance has several issues that can cause misleading results. Especially with correlated features (very common): explained.ai/rf-importance $\endgroup$
    – Jon Nordby
    Commented May 1 at 15:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.