In the very early papers on gradient boosting, the ensemble would include a constant and a sum of base learners i.e.

$F(X) = a_0 + \sum\limits_{i} a_i f_i(X)$

The constant is fitted first (i.e. if the loss is squared error, the $a_0$ will be the unconditional mean of the target)

Somewhere along the way this got dropped, i.e. as far as I can see a number of the current "state of art" gbdt algorithms fit a constant (xgboost, catboost etc). I believe the R mboost package does though.

Why is this, does anyone have any reference where it is discussed (I'm performing regression so particularly interested in this aspect). In particular with a regression tree as a base-learner, can it learn the constant more efficiently than the original algorithms?

One of my motivations is that when I boost with no constant, for a problem where the target is always positive, it seems to converge from below so it almost always has a bias in the residuals unless you have enough data such that you can boost for long enough to fit the highest values of the targets without overfitting (specifically I fit the relationship between temperature and electricity demand).

Edit: Some references

[Prokhorenkova et. al.] "CatBoost: unbiased boosting with categorical features" - Algorithm 1 intialises M to 0. I've used catboost a lot, I can confirm that explicitly setting the starting value to the mean produces a differerent result.

[Chen et. al.] "XGBoost: A Scalable Tree Boosting System" - Equation 1 excludes the constant.

[Sigris] "Gradient and Newton Boosting for Classification and Regression" - there's no mention of the constant alough it's not explicitly excluded

  • $\begingroup$ Can you provide a reference (link or so) to the claim that "none of the current "state of art" gbdt algorithms fit a constant" and a reference on the first claim (constant + sum of base learners). Isn't the constant here the initialization value? So a start value for boosting (often the unconditional mean?) $\endgroup$
    – Peter
    Commented Jul 21, 2019 at 12:35
  • $\begingroup$ I probably shouldn't use None, I've added a couple of references. I've read several others (plus many reproduce Freedmans algorithm which I used above, but simultaneously present the ensemble with no mean). You're correct that it's the initialisation value, maybe the expectation is the user can select if they initialise to the mean or not. But this isn't really what Freedman wrote, he has the constant being the argmin of the loss function, which is the mean iff the loss is RMSE $\endgroup$ Commented Jul 22, 2019 at 0:24
  • $\begingroup$ At least for xgboost: Isn't base_score (for classification), the initialization value (so class mean)? Not sure how this works for regression, but for classification there seems to be a non-zero starting value... stackoverflow.com/questions/47596486/… $\endgroup$
    – Peter
    Commented Jul 22, 2019 at 8:41
  • $\begingroup$ I'm not all that familiar with xgboost but I think you're probably correct, there's something similar in catboost (you pass a vector of starting predictions so you're free to pass the mean, or the result from some previous model if for example you fit multiple models sequentially for whatever reason). I suspect they decided to provide this feature rather than explicitly solving for the constant but not really sure. $\endgroup$ Commented Jul 22, 2019 at 22:56


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.