confused on "real score" vs "decision value" in classification trees

Question

I'm reading the guide to XGBoost and am confused about the distinction it draws between the scoring systems of decision trees and classification/regression trees. The paragraph I am hung up on is:

A CART [classification and regression tree] is a bit different from decision trees, in which the leaf only contains decision values. In CART, a real score is associated with each of the leaves, which gives us richer interpretations that go beyond classification.

I am not at all sure what this means. My understanding of regression decision trees is that each leaf has a value which is the mean label for all training examples that get assigned to that leaf (after following the structure of the tree). In a multi-tree model, when predicting a new example we navigate it through each of the trees and then average the values of the leaves it ends up in; this average is the final prediction.

My questions are:

a) Is my understanding of decision trees correct and

b) What is being done differently in the CART trees used by XGBoost and LightGBM? From the drawing just below the quoted paragraph, it seems like each leaf has a 'prediction score,' which gets summed across the tree and then somehow processed into the final regression prediction?

Answer 1

I'm not sure that these terminologies are universal, but the xgboost documentation appears to be considering a "decision tree" to specifically mean that the predictions made are hard class predictions (the mode of the classes among training data in a leaf), not probability predictions, and therefore not usable for regression tasks.

Regression trees on the other hand generally average the target values in each leaf, and that leads to a useful "soft" classifier version of classification trees as well. Random forests and AdaBoosting may use either hard or soft voting, but gradient boosting requires each learner to be a regressor (fitting to pseudo-residuals), and so XGBoost and LightGBM both use those.

Answer 2

Many alleged "classification" models actually predict probabilities, and then there is some decision function that maps the probability to a category. The common decision function is to take the category that has the highest probability, but you can pick any threshold. You might even choose not to use a threshold and to do direct evaluation of the probability outputs. This gives the richer interpretation that is mentioned, as it allows for, as two examples, risk estimation and calibration.

Frank Harrell (Vanderbilt stat professor) has two blog posts about this that are worth reading.

Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

Classification vs. Prediction

The related Stack, Cross Validated (statistics), tends to talk about this topic more than Data Science. You might be interested in searching for "proper scoring rule" on there. I have several posts on the topic.