In Random Forest, each tree is not fed with the full batch of training data, only a sample.

How does this work for Xgboost? If this sampling happens as well, how does it work for this ML algorithm?

  • $\begingroup$ I understand that you are asking about the sampling techniques used to train each tree. $\endgroup$ – Carlos Mougan Jan 8 at 9:42
  • $\begingroup$ yes, how data is sampled by Xgboost while training the training. $\endgroup$ – Aman Raparia Jan 8 at 9:45
  • $\begingroup$ like in Catboost docs they have mentioned the bootstrap strategy, but I am not very much clear in Xgboost $\endgroup$ – Aman Raparia Jan 8 at 9:48

In Gradient Boosting the simple tree is built for only a randomly selected sub-sample of the full data set (random without replacement).

While on the other hand, Random Forest the samples for each decision tree are selected via bootstrapping; sampling a dataset with replacement.

Particularly for xgboost (see paper here)the ratio of sampling of each tree can be modified with the following hyperparameter:

  • subsample [default=1]

Subsample ratio of the training instances. Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees. and this will prevent overfitting. Subsampling will occur once in every boosting iteration.

There are more parameters such as colsample_bytree, colsample_bylevel, colsample_bynode.

Bootstrapping techniques can vary with the implementation and. Normally I use Catboost implementation for several reasons.

You can read about the sampling of cb that you can use in the documentation here

Moreover, a paper that I found interesting a bit ago is Minimal Variance Sampling in Stochastic Gradient Boosting . I will quote one of the sentences from the abstract

Different sampling approaches were proposed, where probabilities are not uniform, and it is not currently clear which approach is the most effective...it leads to a new sampling technique, which we call Minimal Variance Sampling (MVS)...The superiority of the algorithm was confirmed by introducing MVS as a new default option for subsampling in CatBoost

Here the Catboost development team explains what is the new sampling technique that is the default in their algorithm.

| improve this answer | |
  • 3
    $\begingroup$ N.B., xgboost's subsampling is done without replacement, unlike bootstrapping. $\endgroup$ – Ben Reiniger Feb 6 at 13:15
  • $\begingroup$ @BenReiniger Thanks for the correction! :) Let me know if you see something that can be improved, please $\endgroup$ – Carlos Mougan Feb 6 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.