2
$\begingroup$

I know how to implement linear objective function and linear boosts in XGBoost. My concrete question is: when the algorithm it fits the residual (or the negative gradient) is it using one feature at each step (i.e. univariate model) or all features (multivariate model)?

Any reference to documentation about the linear boosts in XGBoost will be appreciated.

Thanks!

EDIT: Linear boosts can be implemented in XGBoost by setting the 'booster' parameter to 'gblinear'. See: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/ for useful information on linear boosting. Note that I am not speaking about the objective function (which can be also linear) but about the boosts themselve.

$\endgroup$
1
$\begingroup$

Refer to the authors' paper on xgboost here - XGBoost: A Scalable Tree Boosting System.

Equation 3 on page 2 mentions that in each step all the predictors are used to greedily fit the next additive tree. However, since column sub-sampling is also employed to prevent over-fitting, not all features may actually be used for each step, see section 2.3 of the same paper.

Hence, the specific answer to your query is that the algorithm uses all features to fit the residual.

$\endgroup$
  • $\begingroup$ Thanks for the paper! However, equation 2 and section 2 tells about boosting using tree models. I am asking about linear regression boosters... $\endgroup$ – Escachator Aug 18 '16 at 11:04
  • $\begingroup$ Not sure I follow your comment, but XGBoost uses regression trees. If you can, do share a specific example with which your query can be illustrated. $\endgroup$ – Sandeep S. Sandhu Aug 18 '16 at 11:25
  • 1
    $\begingroup$ Sure, if you read this tutorial: ncbi.nlm.nih.gov/pmc/articles/PMC3885826 you will see that Gradient Boosting Machine can use different boosts, like trees or linear regression. The last one can be done with XGBoost by setting the 'booster' parameter to 'gblinear'. My question is how the specific gblinear works in detail. I havre edited the question to add this. Thanks. $\endgroup$ – Escachator Aug 18 '16 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.