2
$\begingroup$

What exactly is the difference between model-based boosting and gradient boosting? For an intro to model-based boosting see https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf It seems to me that both terms are equivalent. However, both are used in various literature...

$\endgroup$
2
$\begingroup$

Gradient Boosting is fitting a base learner $f_{i}(X)$ to the gradient of the loss function of an existing model $F_{i-1}(X)$ i.e. find base learner $f_i$ which minimises $L(-g_i, f_t(x_i))$ where $g_i$ is the gradient of $L(y_i,\hat{y}_i)$ with respect to $\hat{y}=F_{i-1}(X)$ at the current iteration $i$. Effectively it's gradient descent in function space.

Component wise boosting schemes such as that used by mboost have a list of base learners of which one is selected at each step, i.e.

form2 <- y ~ bols(x1) + bols(x2) + bols(x1, by = x2, intercept = FALSE) +
               bspatial(x1, x2, knots = 12, center = TRUE, df = 1)

Specifies 4 possible base learners, bols(x1), bols(x2), bols(x1,by=x2) and bspatial(x1,x2), all of which are regression splines.

More generally gradient boosted decision trees fits a tree at each step. So the base learners are arguably more complex.

I believe the terms model based and functional are equivalent and both `mboost' and GBDT are examples.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I think model-based boosting is a generalization of gradient boosting which allows for more complex base predictors then trees.

Reference: http://www.jmlr.org/papers/volume11/hothorn10a/hothorn10a.pdf

| improve this answer | |
$\endgroup$
  • $\begingroup$ To be honest, I rather think the opposite is true. Model-based boosting is componentwise gradient boosting, i.e. a special case of gradient boosting, where the base learner (of whatever nature) is updatet not as a whole but only componentwise, that is parameter-wise. $\endgroup$ – Julian Rodemann Jul 14 '19 at 12:51
  • $\begingroup$ Gradient boosting does allow for more complex base learners than trees, but it is highly popular in combination with trees. $\endgroup$ – Julian Rodemann Jul 14 '19 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.