What is the difference between Permutation feature importance vs. RandomForest feature importance? What are the disadvantages vs. advantages of the two techniques?
Forest paper "We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories."
This is saying that if a feature varies on its ability to detect based on class it will be fault. This seems to imply the methodology is sensitive to data outliers like maybe a feature if 90% responsible for classification of 5% of the variables it may be the most important feature even though that might not really be accurate.
Scikit-learn "Permutation feature importance is a model inspection technique that can be used for any fitted estimator when the data is rectangular. This is especially useful for non-linear or opaque estimators. The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled 1. This procedure breaks the relationship between the feature and the target, thus the drop in the model score is indicative of how much the model depends on the feature. This technique benefits from being model agnostic and can be calculated many times with different permutations of the feature."
The data being rectangular means it is a multivariate feature array table. The model working in non linear scenarios means it is able to be beneficial in prediction even when outputs follow non linear functions like XOR.