# How does class_weight work in Decision Tree?

I am interested in Cost-Sensitive learning. And I am trying to understand how class_weight in DecisionTree works in terms of math. I read a lot of articles that there are a lot of algorithms Cost Sensitive Decision Tree. So what exactly does class_weight do in Decision Tree?

## 1 Answer

It is used, for example, when classes are imbalanced, so different weights are assigned to different classes, instead of equal ones.

Another case is when some class is more significant than others, so loss wrt this class counts more.

The class_weight parameter (eg for decision tress) is used by giving different weight to different class samples (doc: https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html, src: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/class_weight.py#L11), which is then used to position the sample accordingly.

Note that class_weight can be used in different ways depending on algorithm and model used. What makes sense for each algorithm. Mathematicaly, usually, is a simple multiplication of the sample value in some loss function

The original gini impurity is defined as:

$${\displaystyle \operatorname {I} _{G}(p)=\sum _{i=1}^{J}\left(p_{i}\sum _{k\neq i}p_{k}\right)=\sum _{i=1}^{J}p_{i}(1-p_{i})=1-\sum _{i=1}^{J}{p_{i}}^{2}}$$

If classes are assigned weights $$w_i$$ then the weighted gini impurity is computed as follows:

the weight of all the observations in a potential child node, $$c$$, is

$$t_c = \sum_i w_i * n_i$$

where $$n_i$$ is the number of observations of class $$i$$ in $$c$$, and $$w_i$$ is the weight assigned to class $$i$$.

The impurity of child node $$c$$ is then

$$i_c = 1 - \sum_i (\frac{w_i * n_i}{t_c})^2$$

where $$n_i$$ is again the number of observations of class $$i$$ in the node, $$w_i$$ is the weight assigned to the class and $$t_c$$ is as calculated previously. The impurity of the entire potential split is then

$$\sum_c \frac{t_c}{t_p} * i_c$$

where $$t_c$$ and $$i_c$$ are as calculated previously, and $$t_p$$ is the total weight of all observations in the parent node that is being split.

• Yes, I understand that. But I have a question how does it look from the mathematical point of view. In what formula are these weights used? Mar 21, 2021 at 10:54
• See updated answer Mar 21, 2021 at 11:12
• Thank you. Could you help me and write how it looks in CART algorithm? (DecisionTree with criterion='gini') Mar 21, 2021 at 11:41
• I am not aware of that, either you should look up the reference paper or an implementation source code. Although I do not expect much difference from what is stated in answer Mar 21, 2021 at 11:48
• OK, thank you. I just read a lot of different articles, and there are many modifications to weighted decision trees. And I wanted to know what is implemented in python Mar 21, 2021 at 11:58