Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the prediction is their average: $\frac1nT_1(\underline{x}_i)+...+\frac1nT_n(\underline{x}_i)$
In the book:
https://www.amazon.com/Ensemble-Methods-Data-Mining-Predictions/dp/1608452840
the authors have suggested that post-processing Adaboost with linear regression, they have witnessed improved performance across the board. Namely, they suggest doing linear regression with the new training data:
$((T_1(\underline{x}_1),...,T_n(\underline{x}_1),y_1),...,((T_1(\underline{x}_N),...,T_n(\underline{x}_N),y_N)$
and then use the resulting coefficients rather than the coefficients $1/n$ for the different trees.
This book was authored way back in 2010, and XGBoost, so far as I can tell, does allow this post-processing. So I was wondering what's going on:
- Is it that this method was largely found to be unhelpful?
- Is it that this method is not well known enough yet?
- Or -- is this method used all the time and I just don't know it?