1
$\begingroup$

All of the popular evaluation metrics (ROC-AUC, Confusion Matrices, etc.) require two lists as parameters: a list of the actual y labels associated with some arbitrary group of training examples (x's), and a parallel list of the predicted labels given to those x's by the model.

To construct such lists, you must separate a testing/validation data set from the training set you feed the model on. However, random forests automatically partition 1/3 of the training set you give it to calculate the out of bag score. I don't believe you can stop the bagging process that causes this, since I think it is vital to how the random forest functions. Because the RF model never sees 1/3 of the training set (due to bagging), do RFs create a less thorough image of the dataset then, say, a neural network would whenever a testing set is reserved for evaluation?

|improve this question
$\endgroup$
  • $\begingroup$ What does a "less thorough image" mean? How wuld you measure it? $\endgroup$ – D.W. Jun 22 '18 at 22:42
  • $\begingroup$ By less thorough, I mean it considers less of the initial training dataset, because you partition off some data for oob, and some more data for testing. Doing this would lead you to optimize a model one a less complete picture of the data. I guess you could measure it based on the final performance of the model compared to other types of models. Thanks for commenting. $\endgroup$ – H Froedge Jun 22 '18 at 23:48
  • 1
    $\begingroup$ Please don't clarify in the comments. Instead, edit the question so it reads well for someone who encounters it for the first time, and so people don't need to read the comments. I don't see why you care whether it "considers less of the training set"; if it's just as accurate, who cares how it managed to achieve that? Perhaps you really mean to ask "are RFs less accurate than neural networks because of bagging"? $\endgroup$ – D.W. Jun 22 '18 at 23:54
1
$\begingroup$

The out-of-bag-error is calculated from the samples that are not used anyway for the particular tree. The original set of records gets bootstrapped. So, a new set is generated that does not contain all samples anyway. The out-of-bag set can then be used to monitor the performance of the model. When the out-of-bag-error goes up (given it has a significant size) it means that the current tree is over-fitting the training sample. So, out-of-bag sampling can be used to prevent over-fitting and therefore makes the model more robust rather than less robust.

So, in fact all samples are used to train the random forest model. Though each tree only uses a subset of the dataset at a time. Don't confuse the RF model with the individual trees!

Look here for a more detailed description.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.