2
$\begingroup$

I'm reading the guide to XGBoost and am confused about the distinction it draws between the scoring systems of decision trees and classification/regression trees. The paragraph I am hung up on is:

A CART [classification and regression tree] is a bit different from decision trees, in which the leaf only contains decision values. In CART, a real score is associated with each of the leaves, which gives us richer interpretations that go beyond classification.

I am not at all sure what this means. My understanding of regression decision trees is that each leaf has a value which is the mean label for all training examples that get assigned to that leaf (after following the structure of the tree). In a multi-tree model, when predicting a new example we navigate it through each of the trees and then average the values of the leaves it ends up in; this average is the final prediction.

My questions are:

a) Is my understanding of decision trees correct and

b) What is being done differently in the CART trees used by XGBoost and LightGBM? From the drawing just below the quoted paragraph, it seems like each leaf has a 'prediction score,' which gets summed across the tree and then somehow processed into the final regression prediction?

$\endgroup$
3
  • $\begingroup$ Remember how logistic regression predicts a probability and not just a classification? $\endgroup$
    – Dave
    Sep 13 at 18:30
  • $\begingroup$ My understanding is that the value of the leaf gets put into some S function to determine a final probability. Is that what is happening in regression too, there exists some other function to convert the leaf value to a final output? $\endgroup$
    – Sinnombre
    Sep 14 at 16:45
  • $\begingroup$ The leaf values are not passed through any link function, but their sum is, to produce final predictions. For ordinary regression the link is the identity function. $\endgroup$ Sep 14 at 19:47
1
$\begingroup$

I'm not sure that these terminologies are universal, but the xgboost documentation appears to be considering a "decision tree" to specifically mean that the predictions made are hard class predictions (the mode of the classes among training data in a leaf), not probability predictions, and therefore not usable for regression tasks.

Regression trees on the other hand generally average the target values in each leaf, and that leads to a useful "soft" classifier version of classification trees as well. Random forests and AdaBoosting may use either hard or soft voting, but gradient boosting requires each learner to be a regressor (fitting to pseudo-residuals), and so XGBoost and LightGBM both use those.

$\endgroup$
2
  • $\begingroup$ I am specifically talking about regression here. Are you saying the description in the documentation only pertains to probability calculations, and that in regression XGBoost and LightGBM both use the 'mean of members of the leaf' method I mention? $\endgroup$
    – Sinnombre
    Sep 14 at 16:47
  • $\begingroup$ The excerpt you quote is about two different methods: "decision trees" which are strictly for classification and only produce a class as output, and "CART", which produces something more continuous, and is used by xgb and lgbm. Gradient boosting cannot work with a hard-class "decision tree". $\endgroup$ Sep 14 at 19:34
-1
$\begingroup$

Many alleged "classification" models actually predict probabilities, and then there is some decision function that maps the probability to a category. The common decision function is to take the category that has the highest probability, but you can pick any threshold. You might even choose not to use a threshold and to do direct evaluation of the probability outputs. This gives the richer interpretation that is mentioned, as it allows for, as two examples, risk estimation and calibration.

Frank Harrell (Vanderbilt stat professor) has two blog posts about this that are worth reading.

Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

Classification vs. Prediction

The related Stack, Cross Validated (statistics), tends to talk about this topic more than Data Science. You might be interested in searching for "proper scoring rule" on there. I have several posts on the topic.

$\endgroup$
1
  • $\begingroup$ Please explain the downvote. $\endgroup$
    – Dave
    Oct 12 at 0:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.