What is the difference between Permutation feature importance vs. RandomForest feature importance? What are the disadvantages vs. advantages of the two techniques?

  • $\begingroup$ If you need any clarification just ask. $\endgroup$ Nov 12 '19 at 0:34


Forest paper "We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories."

This is saying that if a feature varies on its ability to detect based on class it will be fault. This seems to imply the methodology is sensitive to data outliers like maybe a feature if 90% responsible for classification of 5% of the variables it may be the most important feature even though that might not really be accurate.


Scikit-learn "Permutation feature importance is a model inspection technique that can be used for any fitted estimator when the data is rectangular. This is especially useful for non-linear or opaque estimators. The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled 1. This procedure breaks the relationship between the feature and the target, thus the drop in the model score is indicative of how much the model depends on the feature. This technique benefits from being model agnostic and can be calculated many times with different permutations of the feature."

The data being rectangular means it is a multivariate feature array table. The model working in non linear scenarios means it is able to be beneficial in prediction even when outputs follow non linear functions like XOR.

  • $\begingroup$ I still don't understand what is means by "potential predictor variables vary in their scale of measurement". Is it talking about standardization? Could you elaborate more on when RF based feature importance fail in detail? $\endgroup$
    – Eric Kim
    Nov 12 '19 at 6:05
  • $\begingroup$ @EricKim it means that RF feature importance can be biased towards numerical features compared to categorical features (same is true for categorical features of high cardinality, i.e. a high number of unique values in the dataset, compared to those w/ lower cardinality). $\endgroup$
    – Sammy
    Jul 13 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.