4
$\begingroup$

I have a random forest classifier and Multinomial Naive Bayes. For feature importance, I used gini index for random forest and for Multinomial Naive Bayes I used the coefficients of each feature. Then normalized to compare the two lists but there is a big difference between the two. Is this normal? Something like

RF vs NB

  1. A - C
  2. B - D
  3. C - A
  4. D - B
Improve this question
$\endgroup$
6
$\begingroup$

Is this normal?

It is not surprising.

First, you are using different measures of feature importance. It’s like measuring the importance of people (or simply sorting them) using their a) weight, b) height, c) wealth and d) IQ. With a and b you might get quite similar results, but these results are likely to be different from results obtained with c and d.

Second, the performance of your models is likely to be different. In extreme case, output of one of your models could be completely rubbish (in your case it is more likely to be NB). Then the feature importance metrics produced with such model is not credible. In less extreme scenarios when the difference in models‘ performance is not so dramatic the trustworthiness of importances produced by two different models is more comparable. Still the importances might be quite different due to the first argument, i.e. different language used to capture the importance.

Edit:

You have not asked about it in your question, but there are feature importance approaches which are model agnostic, i.e. can be applied to any predictive model. For example, check the permutation importance approach, described in chapter "15.3.2 Variable Importance" in The Elements of Statistical Learning

Improve this answer
$\endgroup$
7
  • $\begingroup$ Why do you think that NB is not correct? $\endgroup$ – Thomas Lee Mar 8 '18 at 20:47
  • $\begingroup$ It was just an extreme example, the closer the performance the more credible importances you could get. Btw, what is your AUC on validation dataset for RF and NB? $\endgroup$ – aivanov Mar 8 '18 at 22:13
  • $\begingroup$ AUC RF = 0.98 and AUC NB =0.84 $\endgroup$ – Thomas Lee Mar 9 '18 at 1:51
  • $\begingroup$ Thank you. It's quite a gap, thus you probably should trust more the feature importances computed with the RF model. $\endgroup$ – aivanov Mar 9 '18 at 9:51
  • $\begingroup$ A more general way of comparing feature importance between two different techniques may be comparing the relative rankings - i.e. feature importance rank in NB v feature importance rank in RF. $\endgroup$ – bradS Mar 9 '18 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.