I am interested in Cost-Sensitive learning. And I am trying to understand how class_weight in DecisionTree works in terms of math. I read a lot of articles that there are a lot of algorithms Cost Sensitive Decision Tree. So what exactly does class_weight do in Decision Tree?


It is used, for example, when classes are imbalanced, so different weights are assigned to different classes, instead of equal ones.

Another case is when some class is more significant than others, so loss wrt this class counts more.

The class_weight parameter (eg for decision tress) is used by giving different weight to different class samples (doc: https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html, src: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/class_weight.py#L11), which is then used to position the sample accordingly.

Note that class_weight can be used in different ways depending on algorithm and model used. What makes sense for each algorithm. Mathematicaly, usually, is a simple multiplication of the sample value in some loss function

Related: How does the class_weight parameter in scikit-learn work?.

For CARTs with Gini Criterion:

The original gini impurity is defined as:

$${\displaystyle \operatorname {I} _{G}(p)=\sum _{i=1}^{J}\left(p_{i}\sum _{k\neq i}p_{k}\right)=\sum _{i=1}^{J}p_{i}(1-p_{i})=1-\sum _{i=1}^{J}{p_{i}}^{2}}$$

If classes are assigned weights $w_i$ then the weighted gini impurity is computed as follows:

the weight of all the observations in a potential child node, $c$, is

$$t_c = \sum_i w_i * n_i$$

where $n_i$ is the number of observations of class $i$ in $c$, and $w_i$ is the weight assigned to class $i$.

The impurity of child node $c$ is then

$$i_c = 1 - \sum_i (\frac{w_i * n_i}{t_c})^2$$

where $n_i$ is again the number of observations of class $i$ in the node, $w_i$ is the weight assigned to the class and $t_c$ is as calculated previously. The impurity of the entire potential split is then

$$\sum_c \frac{t_c}{t_p} * i_c$$

where $t_c$ and $i_c$ are as calculated previously, and $t_p$ is the total weight of all observations in the parent node that is being split.

Reference: How is the Weighted Gini Criterion defined?

  • $\begingroup$ Yes, I understand that. But I have a question how does it look from the mathematical point of view. In what formula are these weights used? $\endgroup$ – Marni Mar 21 at 10:54
  • $\begingroup$ See updated answer $\endgroup$ – Nikos M. Mar 21 at 11:12
  • $\begingroup$ Thank you. Could you help me and write how it looks in CART algorithm? (DecisionTree with criterion='gini') $\endgroup$ – Marni Mar 21 at 11:41
  • $\begingroup$ I am not aware of that, either you should look up the reference paper or an implementation source code. Although I do not expect much difference from what is stated in answer $\endgroup$ – Nikos M. Mar 21 at 11:48
  • $\begingroup$ OK, thank you. I just read a lot of different articles, and there are many modifications to weighted decision trees. And I wanted to know what is implemented in python $\endgroup$ – Marni Mar 21 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.