What exactly is the difference between model-based boosting and gradient boosting? For an intro to model-based boosting see https://cran.r-project.org/web/packages/mboost/vignettes/mboost_tutorial.pdf It seems to me that both terms are equivalent. However, both are used in various literature...
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
Gradient Boosting is fitting a base learner $f_{i}(X)$ to the gradient of the loss function of an existing model $F_{i-1}(X)$ i.e. find base learner $f_i$ which minimises $L(-g_i, f_t(x_i))$ where $g_i$ is the gradient of $L(y_i,\hat{y}_i)$ with respect to $\hat{y}=F_{i-1}(X)$ at the current iteration $i$. Effectively it's gradient descent in function space.
Component wise boosting schemes such as that used by mboost have a list of base learners of which one is selected at each step, i.e.
form2 <- y ~ bols(x1) + bols(x2) + bols(x1, by = x2, intercept = FALSE) +
bspatial(x1, x2, knots = 12, center = TRUE, df = 1)
Specifies 4 possible base learners, bols(x1), bols(x2), bols(x1,by=x2) and bspatial(x1,x2), all of which are regression splines.
More generally gradient boosted decision trees fits a tree at each step. So the base learners are arguably more complex.
I believe the terms model based and functional are equivalent and both `mboost' and GBDT are examples.
$\begingroup$
$\endgroup$
I think model-based boosting is a generalization of gradient boosting which allows for more complex base predictors then trees.
Reference: http://www.jmlr.org/papers/volume11/hothorn10a/hothorn10a.pdf
-
$\begingroup$ To be honest, I rather think the opposite is true. Model-based boosting is componentwise gradient boosting, i.e. a special case of gradient boosting, where the base learner (of whatever nature) is updatet not as a whole but only componentwise, that is parameter-wise. $\endgroup$ – Julian Rodemann Jul 14 '19 at 12:51
-
$\begingroup$ Gradient boosting does allow for more complex base learners than trees, but it is highly popular in combination with trees. $\endgroup$ – Julian Rodemann Jul 14 '19 at 12:58
