How does XGBoost compute the probabilities in predict_proba()?

Question

I'm using the sklearn wrapper for XGBoost. I didn't manage to find a clear explanation for the way the probabilities given as output by predict_proba() are computed.

In random forest, for example, I understand it reflects the mean of proportions of the samples belonging to the class among the relevant leaves of all the trees.

However, in XGBoost I couldn't understand the computation from the documentation or the code. Shouldn't it give different weights for each tree?

Answer 1

XGBoost is a gradient-boosting algorithm, which means it builds an ensemble of weak decision trees in a sequential manner, where each tree learns to correct the mistakes of the previous trees. To compute the probabilities of each class for a given input instance, XGBoost averages the predictions of all the trees in the ensemble.

The probabilities output by the predict_proba() method of the XGBoost classifier in scikit-learn are computed using the logistic function. Specifically, the predicted probability for a given class is computed as follows:

$$p(y = k | x) = \frac{1}{1 + \exp(-f(x))}$$

where

• $f(x)$ is the predicted value of the classifier for the input sample
• $x$ and $k$ are the index of the class.

The predicted value $f(x)$ is computed as the weighted sum of the predictions of all the individual trees in the XGBoost model, where the learning algorithm determines the weights during training.

This means that each tree in the model contributes to the final predicted probability. Still, the relative importance of each tree may vary depending on the specific values of the input sample and the trained model.

The predicted probability of a class for a given input instance is computed as follows:

1. For each tree in the ensemble, compute the predicted probability of the instance belonging to the class using a sigmoid function, which is a logistic function that maps the output of the decision tree to a probability value between 0 and 1.

2. Average the predicted probabilities of the instance belonging to the class over all the trees in the ensemble.

3. Compute the predicted probability of the instance belonging to the other class as 1 minus the predicted probability of the instance belonging to the class.

4. Return the predicted probabilities of each class as a list or array of two values, where the first value corresponds to the probability of the instance belonging to the first class and the second value corresponds to the probability of the instance belonging to the second class.

In contrast, in a random forest classifier, the predicted probabilities are computed as the mean of the proportions of samples belonging to each class among the relevant leaves of all the trees. This means that each tree in the random forest model contributes equally to the predicted probabilities, and each tree's relative importance is not considered.