Classification using xgboost - predictions

I was trying to build a 0-1 classifier using xgboost R package. My question is how predictions are made? For example in random forests, trees "vote" against each option and the final prediction is based on majority. As regard xgboost, the regression case is simple since prediction on whole model is equal to sum of predcitions for weak learners (boosted trees), but what about classification?

Does xgboost classifier works the same as in the random forest (I don't think so, since it can return predictive probabilities, not class membership).

• Welcome to the site :) Random forests is one of the many ways XGBoost does classification. Can you please make your question more clear?
– Dawny33
Jan 20 '16 at 5:45
• OK, suppose I have xgboost classifier (objective="binary:logistic", metric = "auc") based on 50 trees and new observation based on which I want to make a prediction. How does it work in practice? I see 2 options: Each tree "says" what's the predicted y (depend on the leaf structure where my observation fell into) and the final y is chosen based on majority votes. Or maybe each tree give you predicted probability and final results is an average (not sum like in xgboost regression model) of them. Or maybe there are some more alternatives? Jan 20 '16 at 9:23

The gradient boost algorithm create a set of decision tree.

The prediction process used here use these steps:

• for each tree, create a temporary "predicted variable", applying the tree to the new data set.
• use a formula to aggregate all these tree. Depending on the model:
• bernoulli: 1/(1 + exp(-(intercept + SUM(temporary pred))))
• poisson, gamma: exp(intercept + SUM(temporary pred))
• adaboost: 1 /(1 + exp(-2*(intercept + SUM(temporary pred))))

The temporary "predicted variable" is a probability, having no sense by its own.

The more tree you have, the more smooth is your prediction.( as for each tree, only a finite set of value is spread across your observations)

The R process is probably optimised, but it is enough to understand the concept.

In the h2o implementation of the gradient boost, the output is a flag 0/1. I think the F1 score is used by default to convert probability into flag. I'll do some search/test to confirm that.

In that same implementation, one of the default output for a binary outcome is a confusion matrix, which is a great way to assess your model ( and open a whole new bunch of interrogations).

The intercept is "the initial predicted value to which trees make adjustments". Basically,just an initial adjustment.

• Thanks for the response. Can you elaborate little more on what you wrote? How exactly the intercept is being calculated? I'm referring to tutorial written by the author of xgboost package (xgboost.readthedocs.org/en/latest/model.html - section Tree boosting - additive training) and there is no intercept - just a linear combination of single trees' prediction (I know that this formula refers to regression, but still there is some inconsistency). Last but not least, are the temporary predicted variables 0-1 or there are predicted probabilities stemmed from single trees? Jan 20 '16 at 10:57
• I have some doubts on what you said. For bernoulli and trees that give you some small probability (like 0.05 for all), the more weak learners you have the final predictions converge to 1 (since you add >0 elements in denominator). In that case you would rather guess that 0 is a good prediction. Apart from that, I will definetely peruse documentation you mentioned but is it somehow connected with xgboost package (which is similar to gbm but not exactly). Jan 20 '16 at 14:18
• Not sure to understand, but each time you create a model, all the set of tree is modified. So in the end, there is no similarities between a model with n trees and a model with n+1 trees. Yes, the first one is for gbm. But the concept is equivalent for xgboost.
– YCR
Jan 20 '16 at 15:16
• Actually not. Look at slides 20-21 of homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf. In consecutive steps, new tree is built to "upgrade" current prediction which is constant (once i-th tree is built its form never changes till the end of algorithm). I'm pretty sure that is, by and large, the principle of boosting, isn't it? Jan 20 '16 at 15:33
• At each step, of the creation of one model, yes, the previous tree is kept. But when you create a whole new model, modifying the parameter "number of tree", the two resulting tree may be very different, if only due to the seed used. And other point, the temporary "predicted variable" could be negative.
– YCR
Jan 20 '16 at 15:46