Why is it wrong to use the residuals from one model to fit another? Is it wrong? Is the difference down to how the residuals are used? For example, the residuals are useful features for extending a model's predictive performance, but fitting a model on residuals from another does not explain casual or statistical relationships within the dataset.
I don't know if this is a debate between the differences between machine learning/data science and statistics. Still, I came across this opinion in both the accompanying lecture (linked timestamp for his comments) and the textbook from Richard McElreath  (excerpt below, maintaining the original emphasis and cited reference).
Rethinking: Residuals are parameters, not data. There is a tradition, especially in parts of biology, of using residuals from one model as data in another model. For example, a biologist might regress brain size on body size and then use the brain size residuals as data in another model. This procedure is always a mistake. Residuals are not known. They are parameters, variables with unobserved values. Treating them as known values throws away uncertainty. The right way to adjust for body size is to include it in the same model, preferably a model designed in light of an explicit causal model.
I have seen examples of using residuals from one model to fit another within a machine learning context, such as Regression-Enhanced Random Forests . I think the spirit is generally applied for time series analysis when you detrend the line .
 McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. CRC press. Page 137.
 Zhang, H., Nettleton, D., & Zhu, Z. (2019). Regression-enhanced random forests. arXiv preprint arXiv:1904.10416. Link
 Modelling currently infected cases of covid-19 using h2o driverless ai. (2020, March 30). Retrieved March 02, 2021, from https://www.h2o.ai/blog/modelling-currently-infected-cases-of-covid-19-using-h2o-driverless-ai/
 Freckleton, R. P. (2002). On the misuse of residuals in ecology: regression of residuals vs. multiple regression. Journal of Animal Ecology, 71:542–545. Link