0
$\begingroup$

In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as

$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\limits_{k=1}^t \omega(f_k)$.

In the following lines, it is specified that this objective is equal to

$\sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \omega(f_t)+constant$

though I don't really understand this step - do they mean that

$\sum\limits_{k=1}^{t-1} \omega(f_k) = constant$?

if so, why? and why is this a greedy method?

$\endgroup$

1 Answer 1

1
$\begingroup$

do they mean that

$\sum\limits_{k=1}^{t-1} \omega(f_k) = constant$?

Yes! From the paragraph preceding that:

we use an additive strategy: fix what we have learned, and add one new tree at a time.

That is, we're just trying to build the next tree $f_t$, given that $f_1, \dotsc, f_{t-1}$ have already been built, and so $\omega(f_1), \dotsc, \omega(f_{t-1})$ are all already determined. Gradient boosting is greedy in that sense (earlier trees don't try to look ahead to how later trees will fare, nor do later trees attempt to modify earlier trees), but moreso the tree building process is greedy (earlier splits don't try to look ahead to how later splits will fare).

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.