3
$\begingroup$

I have a dataset with 4 categorical features (Cholesterol, Systolic Blood pressure, diastolic blood pressure, and smoking rate). I use a decision tree classifier to find the probability of stroke. I am trying to verify my understanding of the splitting procedure done by Python Sklearn. Since it is a binary tree, there are three possible ways to split the first feature which is either to group categories {0 and 1 to a leaf, 2 to another leaf} or {0 and 2, 1}, or {0, 1 and 2}. What I know (please correct me here) is that the chosen split is the one with the highest information gain.

I have calculated the information gain for each of the three grouping scenarios:

{0 + 1 , 2} --> 0.17

{0 + 2 , 1} --> 0.18

{1 + 2 , 0} --> 0.004

However, sklearn's decision tree chose the first scenario instead of the third (please check the picture).

Can anyone please help clarify the reason for selecting the first scenario? is there a priority for splits that results in pure nodes. thus selecting such a scenario although it has less information gain?

I have added the frequency of each class/feature/category so it would be easy to calculate the Gini indexenter image description here

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

sklearn doesn't know that your feature is categorical; it's treating it as continuous, for which only splits of the form $x \leq \alpha$ are checked, so your second listed split candidate isn't actually a candidate.

In general, sklearn doesn't support categorical variables (yet?), and you'll need to encode it differently (one-hot?) if you want different behavior.

See also
https://datascience.stackexchange.com/a/52103/55122
how to make a decision tree when i have both continous and categorical variables in my dataset?

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the input @ben I used an ordinal encoder as these variables actually represent (low, medium, high) (this is what was recommended in a post). Also, can you please clarify how come the second split candidate (i.e. {0+2, 1}) is not an actual candidate? $\endgroup$
    – GYSHIDO
    Commented Dec 24, 2019 at 15:51
  • $\begingroup$ If the variable is ordinal, then you don't want to split {0,2},{1}! $\endgroup$
    – Ben Reiniger ♦
    Commented Dec 24, 2019 at 15:55
  • $\begingroup$ ...at least, that's the usual way to use ordinal variables. It could happen (and your information gain computations suggest it happens here) that in your train set (or the subset at a node) the lows and highs behave more similarly to each other than to the mediums. So treating the variable as unordered gives your model more capacity; this may improve performance, or it might cause overfitting, depending on whether the variable "really" connects to the target in a monotone way or not. $\endgroup$
    – Ben Reiniger ♦
    Commented Dec 24, 2019 at 18:57
1
$\begingroup$

You mean first to second, not to third?

In any case possible explanation:

What are your parameters in decision tree. For example for different min_samples_split you can expect different GINI values. You got information gain values (very likely) calculated for all of the samples (rows) of your dataset, but thats not how decision tree calculates it (especially when you set this param) (GINI or inf.gain)

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Hello @Noah-Weber , my only parameter is max_depth=3. do you think that the algorithm prioritizes a split leading to a leaf with 0 Gini impurity even when there is a possible split with higher information gain but no pure leaves? $\endgroup$
    – GYSHIDO
    Commented Dec 24, 2019 at 13:47
  • 1
    $\begingroup$ than its default, so you only take 2 for this param $\endgroup$
    – Noah Weber
    Commented Dec 24, 2019 at 13:48
  • 1
    $\begingroup$ value of min_samples_leaf is 2 $\endgroup$
    – Noah Weber
    Commented Dec 24, 2019 at 13:56
  • 1
    $\begingroup$ Do u think that there is no way that I can pinpoint the reason for choosing this selection? is there a prioritization towards the splits resulting in pure leaves? $\endgroup$
    – GYSHIDO
    Commented Dec 24, 2019 at 14:04
  • 1
    $\begingroup$ You can pinpoint it but you need to understand everyparameter in the docu i sent you. For the first thing change min samples leaves to all, than there is also param that can favour not pure leaves. ITs all in documentation and its reproducible $\endgroup$
    – Noah Weber
    Commented Dec 24, 2019 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.