I've been reading the following papers: https://arxiv.org/abs/1810.11363, https://arxiv.org/abs/1706.09516 and https://www.researchgate.net/publication/318030603_Fighting_biases_with_dynamic_boosting. If I understood correctly, CatBoost solves the problem of bias in pointwise gradient estimates:

A prediction model F obtained after several steps of boosting relies on the targets of all training examples. We demonstrate that this actually leads to a shift of the distribution of $F(x_k)$|$x_k$ for a training example $x_k$ from the distribution of $F(x)|x$ for a test example $x$. This finally leads to a prediction shift of the learned model.


All classical boosting algorithms suffer from overfitting caused by the problem of biased pointwise gradient estimates. Gradients used at eachstep are estimated using the same data points the current model was built on. This leads to a shift in the distribution of estimated gradients in any domain of feature space in comparison with the true distribution of gradients in this domain, which leads to overfitting.

and also:

Gradients used at each step are estimated using the same data points that produced the model built so far. This causes an undesirable dependency between the data used for the gradient estimation and the model approximation. One simple consequence is that the residuals estimated on the data points used for training tend to have smaller absolute values than those for unseen data.

My problem is that I don't actually understand the concept of "bias in pointwise gradient estimates", even after reading the definitions above. Could someone explain to me what this problem is and how CatBoost solves it by implementing ordered boosting? enter image description here


1 Answer 1


If I understood correctly the pointwise gradient estimate is the empirical gradient given a point. And the bias come from the estimate is being heavily data dependent. I didn't read the full paper, but the seems like in order to estimate in the algorithm above looks inteasd of use all the same data points in each model $M_n$, the algorithm make a random permutation of the data for each $n$. Looks like reduce the pointwise grandiet bias, but if it solves I ain't sure.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.