1
$\begingroup$

I am interested in finding out how decision trees chose the order in which they split. I understand that splitting is based in information gain. The attribute with the lowest information gain is chosen as the root node.

If I had a data set with columns:

  1. credit standing
  2. age
  3. income,
  4. marriage status

and I was interested in finding out what determines a person to have a good or bad credit status, Am I correct in saying, I calculate entropy and information gain for each of these categorical attributes against the independent attribute i.e what I am investigating (credit standing), and that the calculation with the lowest information gain is chosen as the root node.

For example, if this root node (first split) was Age, Is the entropy and information gain of Age (the new independent attribute) against the remaining attributes (marriage status and income) calculated, and the calculation with the lowest information gain is then chosen as the second split node, and so on?

ie.

information gain:

credit standing vs age    = 0.01
credit standing vs status = 0.1
credit standing vs income = 0.2

Age is chosen as root (first split) node.

then, information gain:

age vs status = 0.2
age vs income = 0.1

income is chosen as second split node.

Am I understanding this correctly?

Improve this question
$\endgroup$
1
  • $\begingroup$ You partition by the attribute offering the highest, not lowest, information gain (entropy reduction). $\endgroup$
    – Emre
    Nov 17, 2017 at 0:27

1 Answer 1

Reset to default
1
$\begingroup$

Here in your Scenario you need to select the one with more Information Gain rather than the least one and the process goes on till you reach the last feature/last node.

Go through this Link. I think your doing it vice versa, I agree with Emre.

In the above link it was explained with an example to decide whether to play tennis or not.

Do let me know if you have any issues.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Link only answers are frowned upon for a variety of reasons. One big one reason is that links rot. Please spend a couple of minutes and at least summarize the story that the link tells. $\endgroup$
    – Stephen Rauch ♦
    Nov 17, 2017 at 3:57
  • $\begingroup$ I understand. The reason why I appended the link is, Information Gain and Entropy was explained with an example. I thought it would be good if she can go through that example for better understanding. $\endgroup$
    – Toros91
    Nov 17, 2017 at 4:00
  • $\begingroup$ Yes, the way to evaluate an answer which has a link is: Does the answer stand on its own without the link. If you read this answer without the link, it simply says that Emre is Right. Well we can assume that Emre is right because he makes a habit of that, so the answer doesn't say anything. If however you were to take some inspiration from that presentation and present an actual answer here, that would be an answer, and would be worthy of upvotes instead deletion. Thanks... $\endgroup$
    – Stephen Rauch ♦
    Nov 17, 2017 at 4:03
  • 1
    $\begingroup$ Cool, Thanks for your suggestion @StephenRauch , will keep that in mind. Edited the answer too $\endgroup$
    – Toros91
    Nov 17, 2017 at 4:05
  • $\begingroup$ @Toros91 this was an extremely useful link, Thank you so much for replying. $\endgroup$
    – Will.S89
    Nov 17, 2017 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.