2
$\begingroup$

I am interested in finding out how decision trees chose the order in which they split. I understand that splitting is based in information gain. The attribute with the lowest information gain is chosen as the root node.

If I had a data set with columns:

  1. credit standing
  2. age
  3. income,
  4. marriage status

and I was interested in finding out what determines a person to have a good or bad credit status, Am I correct in saying, I calculate entropy and information gain for each of these categorical attributes against the independent attribute i.e what I am investigating (credit standing), and that the calculation with the lowest information gain is chosen as the root node.

For example, if this root node (first split) was Age, Is the entropy and information gain of Age (the new independent attribute) against the remaining attributes (marriage status and income) calculated, and the calculation with the lowest information gain is then chosen as the second split node, and so on?

ie.

information gain:

credit standing vs age    = 0.01
credit standing vs status = 0.1
credit standing vs income = 0.2

Age is chosen as root (first split) node.

then, information gain:

age vs status = 0.2
age vs income = 0.1

income is chosen as second split node.

Am I understanding this correctly?

$\endgroup$
1
  • $\begingroup$ You partition by the attribute offering the highest, not lowest, information gain (entropy reduction). $\endgroup$
    – Emre
    Nov 17, 2017 at 0:27

1 Answer 1

1
$\begingroup$

Here in your Scenario you need to select the one with more Information Gain rather than the least one and the process goes on till you reach the last feature/last node.

Go through These Links. I think your doing it vice versa, I agree with Emre.

In the link it was explained with an example to decide whether to play tennis or not.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.