Metrics to evaluate features' importance in classification problem (with random forest)

Question

I want to evaluate the importance of each of the features of a 2000x60 dataset in a classification problem with random forest.

The most widely used ones apparrently are:

• Cross Entropy-Information Gain
• Gini Importance (SkLearn implementation with feature_importances_)
• Mean Squared Error (H2O implementation with h2o.varimp)

I have also found a rather concise overview of some other metrics for variables' importance at random forests at this research paper.

These are the following:

• Altmann
• Boruta
• Permutation
• Recurrent relative variable importance
• Recursive feature elimination
• Vita
• VSURF

Has anyone used these and which one was the most informative for his/her model?

Do you have any other metrics of this kind for variable importance at random forests?

Answer 1

One of the most commonly used methods is permutation feature importance which is decrease in a model score when a single feature value is randomly shuffled. The result is ranked ordered list of features and with each one having a distributional estimates. It works very well with Random Forests.

Answer 2

Microsoft's SHAP values are very interesting and seem to bridge the gaps between LIME, Input Gradients, Global Surrogate, and basic permutation feature importance. In addition to being fairly comprehensible, Slundberg's library is very comprehensive and has got some great visualizations tools for your model.