I've been using random forest to calculate feature importance however I've asked myself why random forest? so my question became what other machine learning models can calculate feature importance? and is there pros and cons between each models?

I've did some research and here are some that I've found

  1. Statistical correlation score -> calculating each X variable with Y. (statistical method)
  2. coefficients in linear models -> Logistic, linear regression, regularizations(L1, L2, elastic net)
  3. Tree based algorithms -> ex: decision tree, RF, boosting algorithms
  4. permutation importance score

I would like to know if there are any other methods.

Thanks in advance!

  • $\begingroup$ shap values also. They are model agnostic $\endgroup$ – Moreno Jan 12 at 2:36

You can calculate a feature importance for any classifier. Take a look at lime or shap

Shap unifies seven different methods (cited from the shap GitHub page):

1 LIME: Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "Why should i trust you?: Explaining the predictions of any classifier." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016.

2 Shapley sampling values: Strumbelj, Erik, and Igor Kononenko. "Explaining prediction models and individual predictions with feature contributions." Knowledge and information systems 41.3 (2014): 647-665.

3 DeepLIFT: Shrikumar, Avanti, Peyton Greenside, and Anshul Kundaje. "Learning important features through propagating activation differences." arXiv preprint arXiv:1704.02685 (2017). 4 QII: Datta, Anupam, Shayak Sen, and Yair Zick. "Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems." Security and Privacy (SP), 2016 IEEE Symposium on. IEEE, 2016.

5 Layer-wise relevance propagation: Bach, Sebastian, et al. "On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation." PloS one 10.7 (2015): e0130140.

6 Shapley regression values: Lipovetsky, Stan, and Michael Conklin. "Analysis of regression in game theory approach." Applied Stochastic Models in Business and Industry 17.4 (2001): 319-330.

7 Tree interpreter: Saabas, Ando. Interpreting random forests. http://blog.datadive.net/interpreting-random-forests/

Depending on the classification algorithm you could calculate the feature importance by using Gini or gain based methods, too ..


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.