3
$\begingroup$

Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the prediction is their average: $\frac1nT_1(\underline{x}_i)+...+\frac1nT_n(\underline{x}_i)$

In the book:

https://www.amazon.com/Ensemble-Methods-Data-Mining-Predictions/dp/1608452840

the authors have suggested that post-processing Adaboost with linear regression, they have witnessed improved performance across the board. Namely, they suggest doing linear regression with the new training data:

$((T_1(\underline{x}_1),...,T_n(\underline{x}_1),y_1),...,((T_1(\underline{x}_N),...,T_n(\underline{x}_N),y_N)$

and then use the resulting coefficients rather than the coefficients $1/n$ for the different trees.

This book was authored way back in 2010, and XGBoost, so far as I can tell, does allow this post-processing. So I was wondering what's going on:

  1. Is it that this method was largely found to be unhelpful?
  2. Is it that this method is not well known enough yet?
  3. Or -- is this method used all the time and I just don't know it?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.