Given training data $(\underline{x}_1, y_1),...,(\underline{x_N}, y_N)$, one can choose a variety of ensemble method for trees. These algorithms output a set of trees $T_1, ..., T_n$, and then the prediction is their average: $\frac1nT_1(\underline{x}_i)+...+\frac1nT_n(\underline{x}_i)$

In the book:


the authors have suggested that post-processing Adaboost with linear regression, they have witnessed improved performance across the board. Namely, they suggest doing linear regression with the new training data:


and then use the resulting coefficients rather than the coefficients $1/n$ for the different trees.

This book was authored way back in 2010, and XGBoost, so far as I can tell, does allow this post-processing. So I was wondering what's going on:

  1. Is it that this method was largely found to be unhelpful?
  2. Is it that this method is not well known enough yet?
  3. Or -- is this method used all the time and I just don't know it?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.