I believe in Random forest we pick random samples of training data with replacement. My question is there still is a possibility that we might leave some data out. What happens to that. Does it not affect the random forest training? Do we use that left over later at some point.


I suppose it is possible that not all samples are selected during training, depending on the parameters you specify (or that are available in the implementation).

Looking at Scikit-Learn's RandomForestClassifier documentation, we can see that there is a bootstrap argument that can be set to False to ensure all data points are used to fit each of the trees. Otherwise, say you pick some arguments to all be 1 (num_estimators, max_depth, min_samples), then not much data would be used at all! Looking through the source code, there doens't seem to be a check that all data was used.

Another classifier, ExtraTrees (Extremely Randomised Trees) is generally designed to used all samples to train each estimator. However the SciKit learn implementation does allow you to disable that and use random bootstrapping, as is default with the other random forest algorithms.

So to answer your question; it seems the unused samples are simply left out!

  • 2
    $\begingroup$ A couple of points to add. First, each tree will miss about $1/e$ of the data. But then, with 1,2,3,4 trees you expect to completely ignore 37%, 14%, 5%, 2% of the data respectively. With 10 trees it's down to 0.0045%, with 100 trees $3\cdot10^{-42}$%. So for most applications, you're not missing out on much. Second, any missed samples will show up in all the out-of-bag (OOB) scoring estimates, so not necessarily completely wasted. $\endgroup$ Apr 26 '19 at 19:35
  • $\begingroup$ That is a good analysis by @BenReiniger of the common case i.e. you don't need to worry about leaving out any samples. I would once again mention that it is possible to choose parameter that would leave out a lot of samples. E.g. by choosing very few shallow trees and deactivating out-of-bag samples for validation accuracies (by default that is the case in sklearn's implementation!). $\endgroup$
    – n1k31t4
    Apr 26 '19 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.