As the title suggests, I have a question regarding the trees produced through the bagging procedure.

Namely, since the bootstrap samples created to fit trees on are independent and identically distributed (iid), are the resulting trees also iid?

In other words, is there any reason there may be correlation between the "bootstrap trees"?


This question may be more suited to CrossValidated. However:

"Bagging" is short for "bootstrap aggregating", meaning that a random sample with replacement is taken from the overall set. The key here is "with". For example, suppose this is your dataset:

{1, 2, 3, 4, 5}

and you are interested in obtaining 4 samples of size 3. You could end up with the following:

{1, 4, 4}

{1, 4, 5}

{2, 3, 4}

{3, 3, 4}

That is, you can have repeated elements within the same bootstrap sample (e.g. the first result, {1, 4, 4}), and each bootstrap sample could contain the same element (e.g. notice the value '4' in each sample). You can see more on this here.

Yes, it is possible for there to be a correlation among the bootstrapped samples, but if you're interested in, for example, a random forest, then the trees are necessarily NOT correlated. If you were to train 100 trees on the same data set, then you could of course end up with correlated trees. However, in a random forest, the trees are uncorrelated. You can see more on this here.

Hope that helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.