I read a lot about random forest and gradient boosting, but I do not know how these two algorithms really work.

For example, see the simple picture about basketball (picture 1) from this link:

How does Random Forest and how does Gradient Boosting work? Has each tree in the random forest different trainings data AND different features?

Sorry about m question but I don‘t find an easy non-mathematical answer.


Random Forest: Build a decision tree.

  1. Sample N examples from your population with replacement (meaning examples can appear multiple times).
  2. At each node do the following
    1. Select m predictors from all predictors
    2. Split based on predictor that performs best via some objective function
    3. Go to the next node, select another m predictors and repeat

Combine all trees as an average or weighted via some scheme.

Gradient Boosting

One key note is that random forest trees essentially indepedent of each other. Boosting algorithms add a certain depedency to the model.

  1. Initialize a model by finding the minimizer of a certain objective function
  2. For each iteration
    1. Compute the partial derivative of -$L(y_{i}, F(x_{i}))$ with respect to $F(x_{i})$ for all i to n
    2. Fit a tree $h_m$ to the the result from above.
    3. solve $\lambda_m =arg min \sum_{i=1}^{n}L(y_{i}, F_{m-1}(x_{i}) + \lambda h_{m}(x_{i})$
  3. Update model via $F_m(x) = F_{m-1} + \lambda_m h_{m} (x)$

According to Mueller in Introduction to machine learning with Python: a guide for data scientists a Random Forest consists in several Decision Trees whose input is a reorganization of the initial datasets. This reorganization is called bootstrap.

Still according to him, each tree will be trained on a feature space subset. During the training each tree will try to fit the data according to its feature subset.

In Gradient Boosting tree (a part of what is called Gradient boosting machine) the idea is to take shallows trees (shallow here means that each tree will have a very thin subset of the feature space) without "boostraping" the dataset. The different trees are set in a series circuit and each tree will correct the previous tree error. You may tune the learning rate between each tree for a better learning.

PS: Gradient Boosting Machine, in general, combines several "simple" models. In Gradient Boosting Tree the simple model are shallow trees etc. You can have a look to this non mathematical article GB from Scratch

It is not a complete answer but I hope it will be completed and still useful


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.