1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base learners to make them compatible with the loss function.

2) It can be computationally infeasible (in the case of MAE, we would have to calculate the median in every split).

However, I don't understand these points, namely:

1) If we just calculate the residual and have the base learner fit on those values, how exactly would that be any more difficult than calculating the gradients and then fitting on those values?

2) Following on the above question, why exactly do we need to calculate the median in the MAE example instead of just getting the residuals?

Perhaps I don't understand the exact mechanism behind optimizing an individual loss function. It would be really great if someone could demonstrate exactly how MAE is done and in which step would using residuals have been infeasible.

Hmmm, I am little perplexed by your question.

In gradient boosting, we do use the residuals. The residuals are the gradients.

You can check my simple implementation of gradient boosting. This is where the magic happens:

def fit(self, X, y):
self.fs = [self.first_estimator]
# step 0
f = self.first_estimator.fit(X, y)
# step 1
for m in range(self.M):
f = self.predict(X)
# step 2
U = self.loss(y, f)
# step 3
g = clone(self.base_estimator).fit(X, U)
# step 4
self.fs.append(g)


You start by a dummy model $f$. Then you create a new model $g$ based on the errors of the existing ensemble $L(f(x),y)$. The code should be pretty straight-forward. Check the algorithm in the wikipedia page for a more formal presentation.

Why am I using residuals and gradients as interchangeable words?

You have discrete mathematics and continuous mathematics.

In neural networks, we use the definition of gradient from continuous mathematics. You have a loss function that is $C_1$ continuous (meaning that the loss must be differentiable at least once).

$$f'(x) = \lim_{h\to0} \frac{f(x+h)-f(x)}{h}$$

$$f'(x) = f(x+1)-f(x)$$

Personally, I think gradient is a misnomer. But it's marketing. Gradient boosting sounds more mathematical and sophisticated than "differences boosting" or "residuals boosting".

By the way, the term boosting already existed when gradient boosting was invented. It was used for AdaBoost, which changes the weights of the observations based on the loss, rather than training in the residuals. AdaBoost only works for weak models (each model must commit errors), while gradient boosting can do resample and work for strong models.

One important difference between gradient boosting (discrete optimization) and neural networks (continuous optimization) is that gradient boosting allows you to work with functions whose derivative is constant.

In gradient boosting, you can use "weird" functions like MAE or the Pinball function. In my code, you can see that you can choose between MSE, MAE and quantile (which is the pinball loss). These latter two do not work very well for neural networks.

The pinball loss is used for quantile predictions, like wind forecasts in energy systems. This is the reason why gradient boosting is much used in these systems.

Small point: many people think that the reason why MAE and Pinball functions work bad for neural networks is because they are not continuous. While it is true they are not continuous when $y=\hat y$, it's very easy to work-around that. This is not the reason why results are so poor with those losses. The reason why they don't work well is because the optimization steps use the gradient (in gradient descent), and the gradient is constant for the entire function $L'(y,\hat y)=1$ or $L'(y,\hat y)=-1$, so the magnitude of the gradient is always 1. The optimization will depend solely on the learning rate heuristic that you are using, which usually will not work out very well.

• In gradient boosting, each weak learner is fit to the negative gradient of a loss function. When the loss function is a squared loss, e.g. $\frac{1}{2}(y - f(x))^2$, the negative gradient wrt. $f(x)$ happens also to be the residual $y-f(x)$. But this isn't always the case for other loss functions. So saying gradient and residual are the same isn't accurate. Jan 3, 2021 at 15:50

I know this is an old question, but it took me a while to get my head around it

The objective at each iteration of the gradient boosting algorithm is to find a base learner which gives the largest improvement to the loss function L, we can write this as follows

Here $$L()$$ is the loss function, $$y_i$$ is the target, $$\hat{y}_i^{(t-1)}$$ is the estimate from the previous iteration, $$\nu$$ is the learning rate and $$f_t(x_i)$$ is the base learner.

Gradient Boosting is a first-order approximation to the equation above, we take a first-order series expansion, the result is; find base learner $$f_t$$ which minimises $$L(-g_i, f_t(x_i))$$ where $$g_i$$ is the gradient of $$L(y_i,\hat{y}_i)$$ with respect to $$\hat{y}$$ at the current iteration $$t$$. In other words, fit each base learner to the -ve gradient of the loss function from the previous iteration.

Hence for a general loss function, gradient boosting requires that we fit the gradients at each step, and for mean squared loss, the gradient just happens to be the residuals so whilst is seems somewhat intuitive it is also slightly misleading to think of gradient boosting as fitting residuals (or even pseudo residuals as the gradients are often named), it is actually gradient descent in function space.

• I think this is the correct answer. It just happens that for square loss the derivative is the residual Jul 20, 2019 at 18:46