1
$\begingroup$

I am new to Machine Learning and I am trying to undrestand the Out of Bag Error in Random Forests and its use.

Let's say that we have a dataset. First we use the whole dataset (without splitting it) to get a Random Forest and its Out of Bag error. Then we split the dataset, train a Neural Network on the training part and test it on the test part of the dataset.

Can I choose between the two models by comparing the Out of Bag error of the random forest with the total test error of the Neural Network ? Does it make sense ?

Improve this question
$\endgroup$
6
  • $\begingroup$ if you are new to machine learning maybe you should start with non-neural networks. $\endgroup$ – Nischay Namdev Sep 3 '20 at 16:17
  • 1
    $\begingroup$ Just do the split in the same way. Separate test set, or multiple ones (cross validation), is good practice also with Random Forest $\endgroup$ – Jon Nordby Sep 3 '20 at 16:24
  • $\begingroup$ @jonnor thanks.. ok what you suggest is clear.. but then how can I make use of the out of bag error of a random forest ? $\endgroup$ – batman Sep 3 '20 at 16:42
  • $\begingroup$ @nnd my question applies also to non-neural networks though.. please advise for this case then $\endgroup$ – batman Sep 3 '20 at 16:44
  • 1
    $\begingroup$ read about model selection scikit-learn.org/stable/model_selection.html $\endgroup$ – Nischay Namdev Sep 3 '20 at 17:22
1
$\begingroup$

We generally rely on sampling to validate the result of our Model.
We make train/test to test a model on a separate unseen dataset.
If we are doing a Hyperparameter tuning, we keep another set as a validation set to validate the result of new hyperparameters.

Random forest builds each new tree on a Bagged sample from the original sample(train data).
Bagging means sampling is done with replacement i.e. you pick one data point, put it back and then pick the next.

In this process, many duplicated data is sampled and many data points are not sampled.
~63% of the data points are selected Read here
The other "not selected" 37% data points is called Out of Bag samples.
Hence, the way Bagging, and RF is designed, we got another set of data to do our validation. It an opportunity to do a level of validation with these samples.

What it means,

  • You are getting 37% data points to validate your model
  • But OOB is not done with the fully grown Ensemble. It is done using all the trees in the random forest ensemble for which the particular data point was omitted during training Read here
  • It's not equivalent to K-Fold or Train/test on fully built RF but it gives a decent idea about the validation error about to come.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.