is it possible to output more than 2 nodes away from a node in a decision tree? if yes, how to do that with sklearn?

Question

usually a decision tree has one root node, some nodes, and some leaves.

lots tutorial illustrate this as something like binary tree.

is it possible more than 2 nodes away from a node in a decision tree?

this image comes from this post

by "more than 2 nodes", i mean there are more than 3 splits (in this case, 3, Low, Med, High) away from the root node.

if it is reasonable in real life application, plz provide an open dataset on which a decision tree would spit more than 2 nodes, and a piece of sklearn code.

Answer 1

It is possible to make more than a binary split in a decision tree. Chi-square automatic interaction detection (CHAID) is an algorithm for doing more than binary splits.

However, scikit-learn only supports binary splits for many reasons. One primary reason to limit to just binary splits is that the library can support as many splitting criteria as possible with the same API. For example, Gini Impurity only supports binary splits.

In practice, only supporting binary splits is not an issue because a series of binary splits can model any number of simultaneous splits.

Answer 2

I think that scikit-learn only implements binary trees. However, you can turn your example into a binary tree so you can use scikit-learn:

Savings == Low
    True => Assets == Low
        True => Bad Risk
        False => Good Risk
    False => Savings == Med
        True => Good Credit Risk
        False => Income <= 30K
            True => Bad Risk
            False => Good Risk

Answer 3

IMO decision trees are - by definition - designed so that the single "best" split is chosen in each step (Introduction to Statistical Learning, Ch. 8.1). I think you need to split on values low, medium, high, in which case a first split would e.g. occur at (low) vs. (medium, high) and a later split would be between (medium, high), whatever gives the best fit.

Single decision trees often do not have a very good predictive capacity (see. Introduction to Statistical Learning, Ch. 8.2). If you are interested in accuracy of prediction, you should go a step further and grow a random forest with "bagging" or even better "boosting" on many trees (or "ensambles of trees"). In this case, many trees are grown, and they all together make a "vote" on how to predict some outcome.

Random Forest in scikit-learn: https://scikit-learn.org/stable/modules/ensemble.html

Prominent boosting methods are e.g. catboost (https://catboost.ai/docs/concepts/about.html) or lightGBM (https://lightgbm.readthedocs.io/en/latest/).

Alternative (non-tree based) models will be able to make a differentiation by three classes without any problem (ISL, Ch. 4). One example is a logistic model (or "logit") of form risk = b0 + b1*savings. In this models you can also calculate marginal effects, telling you, in case someone moves from class A to class B, by how much will the probability of being a "bad risk" change (marginal effects). https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

You can find a bunch of good code directly covering topics mentionned here and discussed in the book "Introduction to Statistical Learning" online: https://github.com/JWarmenhoven/ISLR-python

In summary: If you are interested in prediction, don't stick to simple decision trees, but move on to something else.

Answer 4

if it is reasonable in real life application

From my understanding of the question, Technically, I think it is perfectly reasonable to have a decision tree/forest splitting into 3 or more nodes from the root node. Kindly check this example if you not. In this example, the predictor variable is whether to play tennis on a given day or not depending on how the climate is on that day.

But I'm not sure whether this is practically possible given the tree structure and the splitting strategy involves find the nodes which correspond to the most reduction in entropy in the model or highest gain in information gain.

P.S: If this answer doesn't give you any new information or clarity, please say so. I'll remove it in order not to misguide or anyone.