2
$\begingroup$

Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is necessary for independence of individual trees.

But why can't we just resample as follows: for tree 1, randomly sample $m$ observations without replacement out of the $n$, where $m$ is still large enough (of course, provided that $n$ is large enough in the first place). Then replenish all observations and repeat the resampling for tree 2, and so on.

Even in this case, I'd imagine that the individual subsamples would be independent. So is there an additional reason for resampling with replacement in bagging?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

No, the samples will not be independent, there is possibility the data samples will be skewed.

For example, imagine a class-imbalanced binary problem, once the minority class is already sampled (large possibility that this can happen given $n$ and $m$) then, without replacement, the rest trees will only sample from the majority class which will produce skewed trees.

Some references:

  1. “Bagging based on resampling with and without replacement is equivalent”, is it?

For random forests, generally, the concept of replacement is considered essential. This is because the underlying concept of random forests is bagging to prevent overfitting, i.e., bagging builds an ensemble of estimators trained on data with a high variance (with regard to the training data they have seen).

  1. Why does random forest use sampling with replacement instead of without replacement?

The basic idea of bootstrapping is that you use your sample as a population. And from it you sample repeatedly, with replacement, to build other samples of the same size as your original sample.

Replacement is an integral part of this process because you are trying to create other possible sample distributions which could have come for your original population based on the sample you have.

Improve this answer
$\endgroup$
4
  • $\begingroup$ You misinterpreted the question: your binary classification example proves that. The question is about using replacement within the sampling of a single bag. Not whether or not to try to create disjunctive bags. $\endgroup$
    – Martijn Courteaux
    Nov 14 at 10:04
  • $\begingroup$ @MartijnCourteaux, ok, still answer applies to single bag. Eg a class-imbalanced task would produce same problem for single bag as well. $\endgroup$
    – Nikos M.
    Nov 16 at 7:50
  • $\begingroup$ @MartijnCourteaux the whole point is to sample the whole population (either single bag or any bag). And this can only be achieved with sampling with replacement as linked references point out. $\endgroup$
    – Nikos M.
    Nov 16 at 7:55
  • $\begingroup$ When sampling 100 samples out of a population of a 1000, I don't see an intuitive reason why you would want to perform this sampling WITH replacement. I figured out now that the definition of a bootstrap sample is to sample as many samples as you have elements in the population. So, my original idea of sampling only 100 out of a 1000 is not technically a bootstrap sample. However, sklearn does support subsampling the population like this (see max_samples in BaggingClassifier), and then the question rises: do we also do this with with replacement? There are enough samples to not replace. $\endgroup$
    – Martijn Courteaux
    4 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.