# Use model output as feature to predict model error in boosting

We use data $$(\boldsymbol{x_1}, \boldsymbol{x_2},\ldots, \boldsymbol{x_n},\boldsymbol{y})$$ to improve the predictability of a physical model $$f(x_1,x_2,\ldots,x_n)$$ that was implemented by domain experts.

Let $$\hat{\boldsymbol{y}}=f(\boldsymbol{x_1}, \boldsymbol{x_2},\ldots, \boldsymbol{x_n})$$. Originally, we decided to fit the errors $$\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}$$ with a statistical model, $$g$$, so the improved model shall be $$g(x_1,x_2,\ldots,x_n) + f(x_1,x_2,\ldots,x_n)$$.

Later I found that adding $$\hat{y}$$ as a feature can train a better $$g$$ for fitting the error $$e$$, so the final model was changed to: $$g(x_1,x_2,\ldots,x_n,\hat{y}) + f(x_1,x_2,\ldots,x_n)$$ or simply $$h(x_1,x_2,\ldots,x_n)=g\left(x_1,x_2,\ldots,x_n,f(x_1,x_2,\ldots,x_n)\right) + f(x_1,x_2,\ldots,x_n)$$

Does this approach have any issue? It seems good to me because $$\hat{y}$$ is just an engineered feature for training $$g$$. Posts like https://stats.stackexchange.com/questions/404809/is-it-advisable-to-use-output-from-a-ml-model-as-a-feature-in-another-ml-model also support my view.

• A possible issue is that the distribution of errors is rarely uniform, so in a few cases there might be a huge error which in turn causes an even larger error when the prediction is reused as feature. Depending on the application, this kind of case might matter or not, Commented Aug 19, 2022 at 16:26
• @Erwan Thank u for the comment! Besides the numerics, do you see any fundamental / theoretical issue with this approach? The main reason why I asked is that, in the gradient boosting framework, every new weak learner only uses the original features to fit the residuals. But this is only an algorithmic characteristic, and does not imply we cannot include $\hat{y}$ to train the weak learner right? Commented Aug 19, 2022 at 17:06
• No, I'm not aware there would be any theoretical issue with the approach. It needs to be tested of course, but imho if it works why not? Commented Aug 19, 2022 at 17:38

What do you mean by $$f$$ is a "physical model"? If you mean something like, "Given some $$x$$, domain experts then use their experience/discretion to estimate $$f(x)$$", which you then feed into some statistical model $$g$$, then I see no issue at all here.

(E.g. $$x$$ is some weather data, we then ask some weather experts their thoughts on the chance of rain tomorrow $$f(x)$$, and then use that as features for some machine learning model.)

In fact, that is simply feature engineering. If $$g$$ is a flexible ML model like neural networks, forests, etc, then worst case these features don't contribute anything but should not really degrade performance. If $$g$$ is a rigid statistical model like OLS or something, then you might run into some various model-specific issues like multi-collinearity etc. Hard to say without knowing what $$g$$ is.

Now if $$f$$ is also a statistical model, then you might run into some issues with overfitting. For example, training a random forest to get $$f(x)$$ then using both $$x$$ and $$f(x)$$ as features in a neural network. But you can work around this with some proper cross-validation and data splitting.

• Also is $h(x)$ supposed to be your full model for $y$? Why add $g$ and $f$ together?
• Yes u guessed it right. $f$ is not a weather model but something very close. Both $f$ and $g$ are nonlinear. I am dealing with clients who are not quite literate in machine learning and I believe they would question how dare u include $\hat{y}$ as a feature since it would seem to "contaminate" the statistical model as it would bring along the errors, which is a result from ignoring the fact that the sole purpose of the statistical model is to fit the error. I need some reference to build a stronger case even though the approach is correct. Commented Aug 22, 2022 at 17:24
• Keep in mind that the true $\epsilon$ is not the same as your observed residuals. I would think that "stacking" the models by estimating $y = g(x, f(x)) + \epsilon_g$ directly would work better than this two-stage procedure of fitting the residuals. Including $f(x)$ as a feature is used in residual connections in neural networks, and is also the same logic behind gradient boosting. So there is precedent at least.