3
$\begingroup$

In the category of Gradient Boosting, I find some terms confusing.

I'm aware that XGBoost includes some optimization in comparison to conventional Gradient Boosting.

  • But are Gradient Boosting Machines (GBM) and GBDT the same thing? Are they just different names?

  • Apart from GBM/GBDT and XGBoost, are there any other models fall into the category of Gradient Boosting?

$\endgroup$

1 Answer 1

4
$\begingroup$

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the optimization technique called gradient descent (Boosting Algorithms as Gradient Descent.

Given this, you can boost any kind of model that you want (as far as I know). Moreover in the scikit learn library, gradient boosting, its under the ensemble folder. You can boost any kind of model (linear, svm) its just that decision trees achieve normally great results with this kind of ensemble. Same way that you can do a bagging with any kind of estimator, but if you do it with a decision tree and add a couple technichalities more, you can call it Random Forest.

From scikit learn documentation: GB builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage a regression tree is fit on the negative gradient of the given loss function.

But are Gradient Boosting Machines (GBM) and GBDT the same thing? Are they just different names?

Gradient boosting machines are a kind of ensemble and Gradient Boosting Decision Tree the particular case when a tree is used as the estimator.

Apart from GBM/GBDT and XGBoost, are there any other models fall into the category of Gradient Boosting?

You can use any model that you like, but decision trees are experimentally the best.

"Boosting has been shown to improve the predictive performance of unstable learners such as decision trees, but not of stable learners like support vector machines (SVM)." Kai Ming TingLian Zhu, Springer

$\endgroup$
2
  • $\begingroup$ Great explanation - I have a question though: is the act of fitting subsequent weak estimators on the residuals sequentially considered gradient descent? My understanding of gradient descent involves taking the derivative of the loss space & some step size to "move" towards some local minima. How does this translate to the gradient boosting ensemble method? $\endgroup$ Oct 22, 2020 at 14:01
  • 1
    $\begingroup$ @weareglenn it might be answered in this paper papers.nips.cc/paper/… But it a great appreciation $\endgroup$ Oct 22, 2020 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.