In Random Forest, each tree is not fed with the full batch of training data, only a sample.
How does this work for Xgboost? If this sampling happens as well, how does it work for this ML algorithm?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
In Gradient Boosting the simple tree is built for only a randomly selected sub-sample of the full data set (random without replacement).
While on the other hand, Random Forest the samples for each decision tree are selected via bootstrapping; sampling a dataset with replacement.
Particularly for xgboost (see paper here)the ratio of sampling of each tree can be modified with the following hyperparameter:
- subsample [default=1]
Subsample ratio of the training instances. Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees. and this will prevent overfitting. Subsampling will occur once in every boosting iteration.
There are more parameters such as colsample_bytree, colsample_bylevel, colsample_bynode.
Bootstrapping techniques can vary with the implementation and. Normally I use Catboost implementation for several reasons.
You can read about the sampling of cb that you can use in the documentation here
Moreover, a paper that I found interesting a bit ago is Minimal Variance Sampling in Stochastic Gradient Boosting . I will quote one of the sentences from the abstract
Different sampling approaches were proposed, where probabilities are not uniform, and it is not currently clear which approach is the most effective...it leads to a new sampling technique, which we call Minimal Variance Sampling (MVS)...The superiority of the algorithm was confirmed by introducing MVS as a new default option for subsampling in CatBoost
Here the Catboost development team explains what is the new sampling technique that is the default in their algorithm.