In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger corrected me). Here there are the sampling techniques implemented for Catboost
- Why is Gradient Boosting sampling done without replacement?
- Why would it be worst to sample with replacement?
- Are there any sampling techniques used in GB that are with replacement?
I quote a paper for SGB:
Stochastic Gradient Boosting is a randomized version of standard Gradient Boosting algorithm... adding randomness into the tree building procedure by using a subsampling of the full dataset. For each iteration of the boosting process, the sampling algorithm of SGB selects random s·N objects without replacement and uniformly