How does class_weight work in Decision Tree

Question

The scikit-learn implementation of DecisionTreeClassifier has a parameter as class_weight. As per documentation:

Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one.

and

The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

My understanding is that it should be used in case of imbalanced classes.

Question: How does the DT (classification) algorithm use this parameter while determining the ideal split for a given node? Does it consider some kind of "weighted" majority class instead of simple majority class in a region in the prediction space?

Answer 1

When deciding on a split at a node, the algorithm basically calculates some metric (entropy or gini impurity) for the given node and for the two resulting left and right nodes after the split. Comparing them tells you how much the split would improve the result.

The statistics for the child nodes are weighted by the number of samples in the left and right node, respectively.

When you use sample_weight this adjusts the count and replaces it with the sum of the sample weights. class_weight gives equal sample weights for each sample based on its class according to its class proportion.

For example, the improvement in impurity is calculated as

$$ \frac{N_{parent}}{N_{total}} * (impurity_{parent} - \frac{N_{right}}{N_{parent}} * impurity_{right-child} - \frac{N_{left}}{N_{parent}} * impurity_{left-child}) $$

Without class_weight or sample_weights, the $N$s are just counts. With class_weight you replace them with the corresponding weights.

The idea is the same for entropy, even though calculated differently.

source code