I'm comparing different techniques for feature selection / feature ranking. Two of the techniques under scrutiny are the mutual information (MI) and the information gain (IG) as used in decision trees, i.e. the Kullback-Leibler divergence.

My data (class and features) is all binary.

All sources I could find state, that MI and IG are basically "two sides of the same coin", i.e. that one can be tranformed into the oher via mathematical manipulation. (For example [source 1, source 2])

Yet, when I rank my features using the two measures they do not result in the same ranking order. But if the two measures are equivalent, shouldn't the ranking be the same?

Can someone help me understand why the rankings are different?

  • $\begingroup$ Can you find a documentation to be ensured that the software/library is naming its implementation correctly? Also, you can use a simple calculable example to see if its naming is correct. $\endgroup$
    – Esmailian
    Mar 19 '19 at 11:36

Remind that $I[X;Y]$ is symmetrical, but $KL(P \Vert Q)$ is not. $I[X;Y] = KL(P(X,Y)\Vert P(X)P(Y))$. Take a look if you are not computing $KL(P(X)\Vert P(Y))$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.