As I understand it, a regular linear regression model already minimizes for squared error, which means that it is the theoretical best prediction for this metric. Does xgboost's "reg:linear" objective do something other than minimize squared error?
$\begingroup$
$\endgroup$
7
-
$\begingroup$ Please search before you ask stats.stackexchange.com/questions/232852/… $\endgroup$– PeterCommented Dec 13, 2019 at 20:34
-
$\begingroup$ I'm voting to close this question as off-topic because it is already answered here stats.stackexchange.com/questions/232852/… $\endgroup$– PeterCommented Dec 13, 2019 at 20:35
-
1$\begingroup$ I'm not sure what feature attribute scaling has to do with my question. After reading the answers to that question anyway, I still fail to see if there is any difference between a regular linear regression model and xgboost's "reg:linear" objective. $\endgroup$– Dan JaouenCommented Dec 13, 2019 at 20:38
-
1$\begingroup$ Boosting is conceptually different to OLS. So your question can be answered „no, not the same thing“. Have a look at Ch 10. web.stanford.edu/~hastie/Papers/ESLII.pdf Squared loss is simply an objective function. It is used for many problems $\endgroup$– PeterCommented Dec 13, 2019 at 20:57
-
1$\begingroup$ The idea of boosting is to do better than OLS. Does not always work, but often. OLS is unbiased. Boosting increases bias but reduces variance. Just try on your own. BTW: Catboost or LightGBM are good alternatives to xgboost. $\endgroup$– PeterCommented Dec 13, 2019 at 21:05
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
Linear regression is a parametric model: it assumes the target variable can be expressed as a linear combination of the independent variables (plus error). Gradient boosted trees are nonparametric: they will approximate any* function.
Xgboost deprecated the objective reg:linear precisely because of this confusion. It has been replaced by reg:squarederror, and has always meant minimizing the squared error, just as in linear regression.
So xgboost will generally fit training data much better than linear regression, but that also means it is prone to overfitting, and it is less easily interpreted. Either one may end up being better, depending on your data and your needs.
answered Dec 13, 2019 at 22:46