3
$\begingroup$

I am having a hard time trying to understand the MSE loss function given in the Introduction to Boosted Trees (beware! My maths skills are the equivalent of a very sparse matrix):

$ \begin{split}\text{obj}^{(t)} & = \sum_{i=1}^n (y_i - (\hat{y}_i^{(t-1)} + f_t(x_i)))^2 + \sum_{i=1}^t\Omega(f_i) \\ & = \sum_{i=1}^n [2(\hat{y}_i^{(t-1)} - y_i)f_t(x_i) + f_t(x_i)^2] + \Omega(f_t) + constant \end{split} $

The second equality sign implies that one could easily derive the second equation from the first one, but I cannot see how. My first naïve attempt was to:

  • express $y_i$ as $a$
  • express $(\hat{y}_i^{(t-1)} + f_t(x_i))$ as $b$
  • and then expand $(a-b)^2$

But I wasn't successful. Any help is really appreciated.

$\endgroup$
3
$\begingroup$

I recall I was struggling for some time deriving the second equation. That constant keeps many of your missing elements. Let's break it down using your $(a-b)^{2}$ notation. We will have $a^{2}$, $b^{2}$, and $2ab$:

  • $a^{2}$: $y_{i}$ is constant since it is your true labels/values, thus $a^{2}$ i.e. $y_{i}^{2}$ goes to the constant.
  • $b^{2}$: $y\hat{}_{i}^{t-1}$+$f_{t}^{2}$+$2y\hat{}_{i}^{t-1}f_{t}$

$y\hat{}_{i}^{t-1}$ is constant as it is prediction in a step before $(t-1)$ that we know already, thus goes to the constant term. The other two terms remain as it is.

  • $2ab$: $2y_{i}(y\hat{}_{i}^{t-1}+f_{t}) = -2y_{i}y\hat{}_{i}^{t-1}-2y_{i}f_{t}$. Here also the first term is constant. Only the second term remains.

The rest should be pretty straightforward, just add things that are left and clean it up and you see the second equation comes out beautifully.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.