from XGBoost tutorial, it described:

In this equation $w_j$ are independent with respect to each other, the form $G_j w_j + \frac{1}{2}(H_j+λ)w_j^2$ is quadratic and the best $w_j$ for a given structure $q(x)$

and the best objective reduction we can get is:

$w^∗_j = \frac{−G_j}{H_j+λ}$

$obj^∗=\frac{−1}{2}\sum_{j=1}^T\frac{G^2_j}{H_j+λ} + γT$

So my confusion is:

if we would like to minimize the $obj$ function, clearly, the best way is to set:

$G_j w_j + \frac{1}{2}(H_j + λ) w_j^2 = 0$

thus we would have the best $w_j^*$ is:

$w_j^* = \frac{-2G_j}{H_j + λ}$

which is different with the official explanation.

anyone please help to point out where am I wrong?


Usually to find an optimum you set the derivative of the function equal to 0. In your case that gives

$$ \frac{dG_j}{dw_J} = G_j + (H_j+\lambda)w_j = 0 $$

leading to

$$ w_j = \frac{-G_j}{H_j+\lambda}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.