Use model output as feature to predict model error in boosting

Question

We use data $(\boldsymbol{x_1}, \boldsymbol{x_2},\ldots, \boldsymbol{x_n},\boldsymbol{y})$ to improve the predictability of a physical model $f(x_1,x_2,\ldots,x_n)$ that was implemented by domain experts.

Let $\hat{\boldsymbol{y}}=f(\boldsymbol{x_1}, \boldsymbol{x_2},\ldots, \boldsymbol{x_n})$. Originally, we decided to fit the errors $\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}$ with a statistical model, $g$, so the improved model shall be $g(x_1,x_2,\ldots,x_n) + f(x_1,x_2,\ldots,x_n)$.

Later I found that adding $\hat{y}$ as a feature can train a better $g$ for fitting the error $e$, so the final model was changed to: $$g(x_1,x_2,\ldots,x_n,\hat{y}) + f(x_1,x_2,\ldots,x_n)$$ or simply $$h(x_1,x_2,\ldots,x_n)=g\left(x_1,x_2,\ldots,x_n,f(x_1,x_2,\ldots,x_n)\right) + f(x_1,x_2,\ldots,x_n)$$

Does this approach have any issue? It seems good to me because $\hat{y}$ is just an engineered feature for training $g$. Posts like https://stats.stackexchange.com/questions/404809/is-it-advisable-to-use-output-from-a-ml-model-as-a-feature-in-another-ml-model also support my view.

Answer 1

What do you mean by $f$ is a "physical model"? If you mean something like, "Given some $x$, domain experts then use their experience/discretion to estimate $f(x)$", which you then feed into some statistical model $g$, then I see no issue at all here.

(E.g. $x$ is some weather data, we then ask some weather experts their thoughts on the chance of rain tomorrow $f(x)$, and then use that as features for some machine learning model.)

In fact, that is simply feature engineering. If $g$ is a flexible ML model like neural networks, forests, etc, then worst case these features don't contribute anything but should not really degrade performance. If $g$ is a rigid statistical model like OLS or something, then you might run into some various model-specific issues like multi-collinearity etc. Hard to say without knowing what $g$ is.

Now if $f$ is also a statistical model, then you might run into some issues with overfitting. For example, training a random forest to get $f(x)$ then using both $x$ and $f(x)$ as features in a neural network. But you can work around this with some proper cross-validation and data splitting.