What is the intuitive meaning of "leaf weight" in xgboost

Question

I looked through Tianqi Chen's presentation, but I'm struggling to understand the details of what the leaf weights are, and I would appreciate if someone could help clarify my understanding.

To put the equations into words on the slide "Put into context: Model and Parameters", the predicted value/score (denoted as yhat) is equal to a sum of the K trees of the model, which each maps the attributes to scores. So far so good, I think.

Then in this slide shown below it gives a mock example of a decision tree that calculates the amount that someone likes a computer game X. (Aside: Isn't this a weird example? Who likes a computer game X by an amount 2? What does that even mean? Why not choose an example that has a more concrete, relatable meaning?)

Now this is where I start to get lost. I can tabulate mock data for the mock example, assuming that the model is perfect so that the weights (w1, w2, w3) are equal to the true value. But even that seems weird: what's the difference between the predicted value/score and the weights?

       x_i: "attributes"          y_i, the true score (not yhat_i, which is the predicted score)
                |                                 |
|¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯|¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯|
| Age <15 (x_0) | is male (x_1) | Amount likes the computer game X  |
| 1             | 1             | 2                                 | (Young boy)
| 1             | 0             | 0.1                               | (Young girl)
| 0             | 1             | -1                                | (Old man)
| 0             | 0             | -1                                | (Old woman)
| 0             | 0             | -1                                | (Young woman, older than 15)

My question is: Can someone please share what would be the ACTUAL function f? I assume it's a vector/matrix, but what are the actual numbers? Then my bonus follow up is how would you compute the f for this mock example? I feel it's such a simple question, but I can't figure out the answer. If someone could break this down for me in nitty gritty detail it would be a huge help. Thanks!

Answer 1

I would read the paper (section 2.2 specifically) for a more concise presentation and formal definition of the terms.

You need to define the loss function $\mathcal{L}$ for your problem. Then $g_i$ and $h_i$ are the first and second derivatives, respectively, of $\mathcal{L}$. The optimal weights are then (equation (5) or slide 27). $$ w_j^* = -\dfrac{\sum_{i \in I_j}g_i}{\sum_{i \in I_j}h_i + \lambda} $$ where $\lambda$ is a regularization parameter.

So those weights give you the minimum of (the second order approximation to) $\mathcal{L}$.

Edit:

Since the weights are given in the example the hard work is done (see Figure 1 in the paper). Following the definitions, $f =w_{q(x)}$ and $q(x)$ maps an instance to a leaf node. Putting this together gives: $$ f = \begin{cases} 2 & \mbox{age} < 15 \mbox{ and } \mbox{male} \\ 0.1 & \mbox{age} < 15 \mbox{ and } \mbox{not male} \\ -1 & \mbox{age} \geq 15 \end{cases} $$