2
$\begingroup$

Let's say you have a good-performing regression model, but the end result is not just yet there.

One of the ideas, I came across to improve model performance is to build a second model using the first model's residuals (y-y_pred) on top. This model would suppose to learn for which observations the first model is overpredicting or underpredicting. The results from the second model would then be used together with the first's to improve the prediction.

Has anyone tried to implement this to a regression task? Are there any resources that I miss that have successfully implemented this?

Here are some of the questions I have:

  • Should the second model be trained only on the residuals? or on residuals+features?
  • What should be the overall best practices to implement this in a sklearn pipeline?
$\endgroup$
1
  • $\begingroup$ I really wouldn't do this. This residuals are the error term and it is a good thing if there are mostly normally distributed. There are other methods to "overfit" the regression, although that is not the classical way regression is used. $\endgroup$ Commented May 6, 2022 at 15:39

1 Answer 1

1
$\begingroup$

I think what you are describing can be compared to gradient boosting for regression. There the concept is similar to what you describe -> you iteratively fit a simple base model to your regression task and then try to fit a next one to the errors you have in order to explained the uncertainty of your first model. This is usually done by decision trees and are therefore not directly to be compared to a linear regression but you might find it useful to look deeper into this concept (especially there are already a lot of packages which are implementing it in Python and R)

$\endgroup$
1
  • $\begingroup$ hmm, you are right, it does seem similar task... At least that's something to go from $\endgroup$ Commented Apr 6, 2022 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.