As I understand it, a regular linear regression model already minimizes for squared error, which means that it is the theoretical best prediction for this metric. Does xgboost's "reg:linear" objective do something other than minimize squared error?

  • $\begingroup$ Please search before you ask stats.stackexchange.com/questions/232852/… $\endgroup$ – Peter Dec 13 '19 at 20:34
  • $\begingroup$ I'm voting to close this question as off-topic because it is already answered here stats.stackexchange.com/questions/232852/… $\endgroup$ – Peter Dec 13 '19 at 20:35
  • $\begingroup$ I'm not sure what feature attribute scaling has to do with my question. After reading the answers to that question anyway, I still fail to see if there is any difference between a regular linear regression model and xgboost's "reg:linear" objective. $\endgroup$ – Dan Jaouen Dec 13 '19 at 20:38
  • 1
    $\begingroup$ Boosting is conceptually different to OLS. So your question can be answered „no, not the same thing“. Have a look at Ch 10. web.stanford.edu/~hastie/Papers/ESLII.pdf Squared loss is simply an objective function. It is used for many problems $\endgroup$ – Peter Dec 13 '19 at 20:57
  • 1
    $\begingroup$ The idea of boosting is to do better than OLS. Does not always work, but often. OLS is unbiased. Boosting increases bias but reduces variance. Just try on your own. BTW: Catboost or LightGBM are good alternatives to xgboost. $\endgroup$ – Peter Dec 13 '19 at 21:05

Linear regression is a parametric model: it assumes the target variable can be expressed as a linear combination of the independent variables (plus error). Gradient boosted trees are nonparametric: they will approximate any* function.

Xgboost deprecated the objective reg:linear precisely because of this confusion. It has been replaced by reg:squarederror, and has always meant minimizing the squared error, just as in linear regression.

So xgboost will generally fit training data much better than linear regression, but that also means it is prone to overfitting, and it is less easily interpreted. Either one may end up being better, depending on your data and your needs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.