# How Decision Tree Classifier works? [closed]

In particular i am using SKLearn with class DecisionTreeClassifier.

I would really like to understand how the tree build itself in a simple visual way. Most explanations use virtue signaling math equations, and I am sure there is a pseudo way explaining this.

Any simple article?

1. How the tree is being created? a simple 2 features example.

2. What it does if 2 same training set of features leads to different classes ?

3. What it does if feature1+feature2 trained to output classA, and later we test for only feature1? would it predict A?

4. Do the training Y values have meaning? for instance if Y classes are 1,2,3,4.. , does it look for a "meaning" in their value and it's relation to the numeric X values? or is it just a class without mathematical meaning?

5. How is this concept different from finding a distance between features vectors? who provide better results and when ?

6. What are ways of adjusting the classifier parameters(branches, depth, etc) to improve it?

Hard to find a simple view about how this magic works without virtue signaling equtions.

I think the first issue here is that you don't have a clear understanding of supervised learning in general. Here are a few basics:

• The input for the training stage is a set of labelled instances. A labelled instance is a fixed-size vector of features values (the features must always be in the same order) and its corresponding label (or class).
• The output of training is a model.
• A model can be applied to unlabelled instances to predict their corresponding labels.

Thus some of the questions can be answered in general:

1. two different training sets lead to two different models. They are completely independent.
2. The set of features is always fixed, it must be exactly the same at training and at testing.
3. The goal of a supervised model is to find relations between the features and the labels, in order to predict as accurately as possible the label for any new set of features. But it doesn't know anything about the meaning of either, it's a purely statistical process, i.e. counting how often something happens.
4. Distances between vectors are used in some learning algorithms (e.g. kNN), not in decision trees. Different learning algorithms rely on different methods, they are not fully comparable.

Now a very brief introduction to decision tree learning to answer question 1:

• The algorithm progressively builds the tree starting from the root, one node at a time.
• When creating a node, the algorithm calculates for every feature how much information about the label is gained by knowing the value of the feature. This is calculated on the current training data at this node.
• It selects the feature $$x_i$$ which brings the highest amount of information about the label: the node is made of a condition such as $$x_i$$ == value, leading to two possible children nodes.
• The two children "receive" the part of the training data left after filtering the value: only the instance where $$x_i$$ == value for the node corresponding to true, and only the instances where $$x_i$$ != value for the node corresponding to false.
• Repeat the process of creating nodes until some condition is satisfied. When reaching a leaf, the algorithm simply assigns the majority label for the subset of training data that it "receives".

The last question:

1. There are several variants of the DT learning algorithm (and there can even be variants in their implementation), and each of them has several hyper-parameters which can be adjusted.

Note that mathematical equations are not "virtue signaling", they are needed for people to understand precisely how to implement the method.

• “A labelled instance is a fixed-size vector of features...” – that's not a good description of supervised learning in general. Even if vectors/tensors are used for storage (which, yes, most data scientists do without questioning it), in lots of applications it doesn't make sense to call the vector elements “features”. Instead, learning what the features actually are is often the main challenge. Commented Mar 1, 2022 at 19:50
• @leftaroundabout thanks for your remark, I wrote this quite fast. I changed to "features values", don't hesitate to edit or propose a better phrasing if you have one. Commented Mar 2, 2022 at 17:16

You must know that learning the optimal decision tree is NP-complete. Any (open-source) implementation of decision trees that you find, such as the one you have mentioned, are, in general, information-based greedy approximations to a hard problem (e.g., Entropy gain, Gini index, etc.). Thus, some mathematics is needed in order to fully grasp the entire picture.

1. You must understand the mathematics behind the approximation to visualize how a decision tree is induced because you must answer the question "Feature $$A_1$$ splits better your dataset in a node of the tree than $$A_2$$?"
2. I do not fully understand your question. If you are asking what it would do a trained decision tree on the same training set but with different classes, well then the answer is simple as it would have unexpected behavior. Observe that a fully grown decision tree without pruning (and no regularization) will give 100% accuracy in full-training (i.e., when it gets the same set for the prediction that has been used for training). Therefore, giving the same training set, but with different classes, to a trained decision tree (on the same training set) it would give an accuracy that, in general, is less than 100%.
3. In general, there is no guarantee to obtain the same class.
4. Again, you must understand the mathematics that is under the hood. As an example, Entropy-based decision tree learning is based on the output target variable ($$Y$$ in your case).
5. This is an interesting question. Decision tree learning is a symbolic approach, while distance-based (or, also called, instance-based) learning is a functional approach. Functional methods are known to have better generalization capacity, while symbolic methods are less statistically robust since they represent coarse concepts in numerical domains, but are easier to interpret when interpretation is a concern.
6. Decision trees are notoriously known to overfit data. Therefore, regularization methods such as pre-pruning or post-pruning are used, such as the number of maximum splits, class purity in a leaf node, etc. There is no rule of thumb for setting such parameters, but you can perform grid search, if wanted.