# difference between model-based boosting and gradient boosting

What exactly is the difference between model-based boosting and gradient boosting? For an intro to model-based boosting see https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf It seems to me that both terms are equivalent. However, both are used in various literature...

Gradient Boosting is fitting a base learner $$f_{i}(X)$$ to the gradient of the loss function of an existing model $$F_{i-1}(X)$$ i.e. find base learner $$f_i$$ which minimises $$L(-g_i, f_t(x_i))$$ where $$g_i$$ is the gradient of $$L(y_i,\hat{y}_i)$$ with respect to $$\hat{y}=F_{i-1}(X)$$ at the current iteration $$i$$. Effectively it's gradient descent in function space.

Component wise boosting schemes such as that used by mboost have a list of base learners of which one is selected at each step, i.e.

form2 <- y ~ bols(x1) + bols(x2) + bols(x1, by = x2, intercept = FALSE) +
bspatial(x1, x2, knots = 12, center = TRUE, df = 1)


Specifies 4 possible base learners, bols(x1), bols(x2), bols(x1,by=x2) and bspatial(x1,x2), all of which are regression splines.

More generally gradient boosted decision trees fits a tree at each step. So the base learners are arguably more complex.

I believe the terms model based and functional are equivalent and both $$`$$mboost' and GBDT are examples.

I think model-based boosting is a generalization of gradient boosting which allows for more complex base predictors then trees.

• To be honest, I rather think the opposite is true. Model-based boosting is componentwise gradient boosting, i.e. a special case of gradient boosting, where the base learner (of whatever nature) is updatet not as a whole but only componentwise, that is parameter-wise. Jul 14, 2019 at 12:51
• Gradient boosting does allow for more complex base learners than trees, but it is highly popular in combination with trees. Jul 14, 2019 at 12:58