I believe in Random forest we pick random samples of training data with replacement. My question is there still is a possibility that we might leave some data out. What happens to that. Does it not affect the random forest training? Do we use that left over later at some point.
I suppose it is possible that not all samples are selected during training, depending on the parameters you specify (or that are available in the implementation).
Looking at Scikit-Learn's RandomForestClassifier documentation, we can see that there is a
bootstrap argument that can be set to
False to ensure all data points are used to fit each of the trees. Otherwise, say you pick some arguments to all be 1 (
min_samples), then not much data would be used at all! Looking through the source code, there doens't seem to be a check that all data was used.
Another classifier, ExtraTrees (Extremely Randomised Trees) is generally designed to used all samples to train each estimator. However the SciKit learn implementation does allow you to disable that and use random bootstrapping, as is default with the other random forest algorithms.
So to answer your question; it seems the unused samples are simply left out!