I want to evaluate the importance of each of the features of a 2000x60 dataset in a classification problem with random forest.

The most widely used ones apparrently are:

  • Cross Entropy-Information Gain
  • Gini Importance (SkLearn implementation with feature_importances_)
  • Mean Squared Error (H2O implementation with h2o.varimp)

I have also found a rather concise overview of some other metrics for variables' importance at random forests at this research paper.

These are the following:

  • Altmann
  • Boruta
  • Permutation
  • Recurrent relative variable importance
  • Recursive feature elimination
  • Vita

Has anyone used these and which one was the most informative for his/her model?

Do you have any other metrics of this kind for variable importance at random forests?

  • $\begingroup$ Have you seen github.com/slundberg/shap? A quick intro.: medium.com/civis-analytics/…. $\endgroup$ Commented Aug 31, 2018 at 5:57
  • $\begingroup$ Thank you for your comment. As we can see there are multiple method to something like this. I am more interested to know which ones have been used personally by you or by data scientists in general and which ones performed the best. Have you used any of the aforementioned methods? How did they perform? $\endgroup$
    – Outcast
    Commented Aug 31, 2018 at 10:55

2 Answers 2


One of the most commonly used methods is permutation feature importance which is decrease in a model score when a single feature value is randomly shuffled. The result is ranked ordered list of features and with each one having a distributional estimates. It works very well with Random Forests.


Microsoft's SHAP values are very interesting and seem to bridge the gaps between LIME, Input Gradients, Global Surrogate, and basic permutation feature importance. In addition to being fairly comprehensible, Slundberg's library is very comprehensive and has got some great visualizations tools for your model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.