How does class_weight work in Decision Tree?

Question

I am interested in Cost-Sensitive learning. And I am trying to understand how class_weight in DecisionTree works in terms of math. I read a lot of articles that there are a lot of algorithms Cost Sensitive Decision Tree. So what exactly does class_weight do in Decision Tree?

Answer 1

It is used, for example, when classes are imbalanced, so different weights are assigned to different classes, instead of equal ones.

Another case is when some class is more significant than others, so loss wrt this class counts more.

The class_weight parameter (eg for decision tress) is used by giving different weight to different class samples (doc: https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html, src: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/class_weight.py#L11), which is then used to position the sample accordingly.

Note that class_weight can be used in different ways depending on algorithm and model used. What makes sense for each algorithm. Mathematicaly, usually, is a simple multiplication of the sample value in some loss function

Related: How does the class_weight parameter in scikit-learn work?.

For CARTs with Gini Criterion:

The original gini impurity is defined as:

$${\displaystyle \operatorname {I} _{G}(p)=\sum _{i=1}^{J}\left(p_{i}\sum _{k\neq i}p_{k}\right)=\sum _{i=1}^{J}p_{i}(1-p_{i})=1-\sum _{i=1}^{J}{p_{i}}^{2}}$$

If classes are assigned weights $w_i$ then the weighted gini impurity is computed as follows:

the weight of all the observations in a potential child node, $c$, is

$$t_c = \sum_i w_i * n_i$$

where $n_i$ is the number of observations of class $i$ in $c$, and $w_i$ is the weight assigned to class $i$.

The impurity of child node $c$ is then

$$i_c = 1 - \sum_i (\frac{w_i * n_i}{t_c})^2$$

where $n_i$ is again the number of observations of class $i$ in the node, $w_i$ is the weight assigned to the class and $t_c$ is as calculated previously. The impurity of the entire potential split is then

$$\sum_c \frac{t_c}{t_p} * i_c$$

where $t_c$ and $i_c$ are as calculated previously, and $t_p$ is the total weight of all observations in the parent node that is being split.

Reference: How is the Weighted Gini Criterion defined?