I came across the concept of Information Gain in decision trees.


Where $I(D_p)$ is the information of the parent node and $I(D_{\text{left}})$ & $I(D_{\text{right}})$ the information for the respective children.

While I think I understand the concept clearly, the name makes me a bit confused. So from information theory, we say that "information" could be interpreted as "surprise/randomness". In this case, Information Gain is actually the amount of Information we lose, not gaining, as higher gain indicates less randomness/information.

What do you think?


1 Answer 1


Surprise and randomness aren't the same thing. A signal that contains more surprising information is more informationally valuable than one that contains less surprising information, but that has nothing to do with randomness. This links to the concept of information entropy, which is what I think the cause of the confusion is here. Information entropy is a measure of the uncertainty in a variable's possible outcomes. A variable with a high degree of informational entropy is more likely to return "surprising" information, but it's not inherently "surprising", just highly variable. A lot of non-surprising outcomes are possible alongside the surprising ones. This isn't a measure of how surprising a particular outcome is, but rather of how many possible outcomes can happen. Information Gain reduces Information Entropy by giving you more information about the possible values that variable can take. So if Alice and Bob are both standing outside a building, and I ask where they are each going to go next, there are hundreds of possible options, or a lot of informational entropy. If Bob is blocking the building's only entrance though, then I know Alice can't enter the building. Information about Bob's position has told me something that constrains what Alice can do. This is Information Gain. I have more information about Alice, not less, because of what I saw Bob doing, but the content of that additional information derives from the fact that I now know Alice is capable of doing fewer things, which could be spun as Alice containing "less information", in a certain sense, but that's not particularly useful.

  • $\begingroup$ Yes, but regarding Alice & Bob, that's another definition of information than the one that is used in "Information Theory", more information indicates more bits, which indicates more surprise, that's why I'm confused. $\endgroup$
    – Caj
    Dec 20, 2023 at 19:34
  • $\begingroup$ Well that's what I said there at the end. You could think of the Alice variable as containing less information in an abstract sense, but that isn't very useful. What the Alice variable contains is less entropy. You have gained information about her by looking at another variable, hence information gain. It's a fairly intuitive term but you are overthinking it. You can't gain information about a system without reducing the entropy of that system. That's basically tautological. Entropy is a measure of hidden information. $\endgroup$
    – Jeremiah
    Dec 20, 2023 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.