I've been reading the following papers: https://arxiv.org/abs/1810.11363, https://arxiv.org/abs/1706.09516 and https://www.researchgate.net/publication/318030603_Fighting_biases_with_dynamic_boosting. If I understood correctly, CatBoost solves the problem of bias in pointwise gradient estimates:
A prediction model F obtained after several steps of boosting relies on the targets of all training examples. We demonstrate that this actually leads to a shift of the distribution of $F(x_k)$|$x_k$ for a training example $x_k$ from the distribution of $F(x)|x$ for a test example $x$. This finally leads to a prediction shift of the learned model.
and
All classical boosting algorithms suffer from overfitting caused by the problem of biased pointwise gradient estimates. Gradients used at eachstep are estimated using the same data points the current model was built on. This leads to a shift in the distribution of estimated gradients in any domain of feature space in comparison with the true distribution of gradients in this domain, which leads to overfitting.
and also:
Gradients used at each step are estimated using the same data points that produced the model built so far. This causes an undesirable dependency between the data used for the gradient estimation and the model approximation. One simple consequence is that the residuals estimated on the data points used for training tend to have smaller absolute values than those for unseen data.
My problem is that I don't actually understand the concept of "bias in pointwise gradient estimates", even after reading the definitions above.
Could someone explain to me what this problem is and how CatBoost solves it by implementing ordered boosting?
