1
$\begingroup$

The original question was posted on StackOverflow. However, taking into account the recommendations of Sergey Bushmanov, I'm providing an answer through this medium.

$\endgroup$

1 Answer 1

4
$\begingroup$

As Sergey has pointed out on the original question, going through the following link one finds the reference to the original article. So, we're going to proced by unraveling the components of each formula (explaining each of them) and then doing a proper interpretation of the splitting criterion that Jerome Friedmann proposed.

1. Probabilities $p_{k}$

First, let's consider \begin{align} p_{k} = exp(F_{k}(x)) / \sum_{l = 1} ^{K} \exp(F_{l}(x)), \end{align}

where $F(x)$ is a logistic transformation. The term $p_{k}$ determine an exponential way to consider probabilities to belong to a particular $k$ class, that gives a mayor probability to the events that are more likely to happen and a lesser probability to the ones that are less likely to happen. For example, consider that the values that $F(x)$ could take are 1, 7 and 49, thus

\begin{align} \frac{\exp(49)}{\exp(1) + \exp(7) + \exp(49)} \approx .9999 > \frac{\exp(7)}{\exp(1) + \exp(7) + \exp(49)} \approx 0. \end{align}

In other words, greater values of $F(x)$ are more likely to happen.

2. Weights $w_{l}$

Once we know that $p_{k} \in [0,1)$, we arrive at the definition of the weights

\begin{align} w_{l} = \sum_{i \in R_{l}} w_{l}(x_{i}) = \sum_{i \in R_{l}}p_{k}(x_{i})(1 - p_{k}(x_{i})), \end{align} for the $x$'s in the region $R_{l}$. In the other words, the weights are the sum of the probabilities, that in the given region $l$, every $x_{i}$ belongs to a certain $k$-class. This is fundamental to understand the criteron proposed by Friedman, as we would see next.

3. Least-Squares Improvement Criterion

The least-squares improvement criterion is

\begin{align} i^2(R_{l}, R_{r}) = \frac{w_{l}w_{r}}{w_{l} + w_{r}}(\overline{y_{l}}-\overline{y_{r}})^2 \:\:\:\:\:\:\:\:(1), \end{align}

considering the fact that we would like to split the current region $R$ into two subregions $(R_{l}, R_{r})$. Remember that at the end we would like to do the split as clean as possible, that is to say, the outcomes are close to the average outcome of all $k$ cases at that node. Therefore, this splitting criterion allow us to take the decision not only on how close we're to the desired outcome, but also based on the probabilities of the desired $k$-class that we're going to find in the region $l$ or in the region $r$.

Disclaimer: This is what I understood by reading the original paper by Friedman, any errors should be pointed out. Thanks

$\endgroup$
3
  • $\begingroup$ I was going to create a custom splitter that was focused on differences in the node means when I saw the FriedmanMSE criterion (github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/…). For a pure comparison of means, I'd think the proxy_impurity_improvement method would just calculate the absolute value of diff. Given that the calculation is more complicated, there must be a different intent, right? I don't follow how the probability discussion in your answer relates to what appears to be a weighted comparison of means for regression. Can you clarify? $\endgroup$ Jul 18, 2020 at 7:13
  • $\begingroup$ I'm also getting very similar results when using FriedmanMSE instead of regular MSE. $\endgroup$ Jul 18, 2020 at 7:16
  • $\begingroup$ Scikit-learn's implementation uses the Friedman's improvement criterion for MSE only in regression trees, but the paper and you talk about probabilities of a certain class k, i.e., classification. There seem to be some differences between the two approaches... $\endgroup$ Sep 2, 2022 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.