# confused on "real score" vs "decision value" in classification trees

I'm reading the guide to XGBoost and am confused about the distinction it draws between the scoring systems of decision trees and classification/regression trees. The paragraph I am hung up on is:

A CART [classification and regression tree] is a bit different from decision trees, in which the leaf only contains decision values. In CART, a real score is associated with each of the leaves, which gives us richer interpretations that go beyond classification.

I am not at all sure what this means. My understanding of regression decision trees is that each leaf has a value which is the mean label for all training examples that get assigned to that leaf (after following the structure of the tree). In a multi-tree model, when predicting a new example we navigate it through each of the trees and then average the values of the leaves it ends up in; this average is the final prediction.

My questions are:

a) Is my understanding of decision trees correct and

b) What is being done differently in the CART trees used by XGBoost and LightGBM? From the drawing just below the quoted paragraph, it seems like each leaf has a 'prediction score,' which gets summed across the tree and then somehow processed into the final regression prediction?

• Remember how logistic regression predicts a probability and not just a classification?
– Dave
Sep 13, 2021 at 18:30
• My understanding is that the value of the leaf gets put into some S function to determine a final probability. Is that what is happening in regression too, there exists some other function to convert the leaf value to a final output? Sep 14, 2021 at 16:45
• The leaf values are not passed through any link function, but their sum is, to produce final predictions. For ordinary regression the link is the identity function. Sep 14, 2021 at 19:47

I'm not sure that these terminologies are universal, but the xgboost documentation appears to be considering a "decision tree" to specifically mean that the predictions made are hard class predictions (the mode of the classes among training data in a leaf), not probability predictions, and therefore not usable for regression tasks.

Regression trees on the other hand generally average the target values in each leaf, and that leads to a useful "soft" classifier version of classification trees as well. Random forests and AdaBoosting may use either hard or soft voting, but gradient boosting requires each learner to be a regressor (fitting to pseudo-residuals), and so XGBoost and LightGBM both use those.

• I am specifically talking about regression here. Are you saying the description in the documentation only pertains to probability calculations, and that in regression XGBoost and LightGBM both use the 'mean of members of the leaf' method I mention? Sep 14, 2021 at 16:47
• The excerpt you quote is about two different methods: "decision trees" which are strictly for classification and only produce a class as output, and "CART", which produces something more continuous, and is used by xgb and lgbm. Gradient boosting cannot work with a hard-class "decision tree". Sep 14, 2021 at 19:34

Many alleged "classification" models actually predict probabilities, and then there is some decision function that maps the probability to a category. The common decision function is to take the category that has the highest probability, but you can pick any threshold. You might even choose not to use a threshold and to do direct evaluation of the probability outputs. This gives the richer interpretation that is mentioned, as it allows for, as two examples, risk estimation and calibration.