The existing answers are quite good but here's more detail. In the paper where he invented the random forest, Breiman touts the out-of-bag calculation as an alternative to cross-validation:
Therefore, using the out-of-bag error estimate removes the need for a
set aside test set.
In the same paper, he doubles down, saying that OOB can be preferable to CV:
...unlike cross-validation, where bias is present but its extent
unknown, the out-of-bag estimates are unbiased
Here's a more concrete example. Let's train a random forest based on this dataset from Kaggle.
col_factor <- readr::col_factor
telco_raw <- read_csv(
col_types = cols(
Churn = col_factor(levels = c("Yes",
Dependents = col_factor(levels = c("Yes",
PaperlessBilling = col_factor(levels = c("Yes",
Partner = col_factor(levels = c("Yes",
PhoneService = col_factor(levels = c("Yes",
SeniorCitizen = col_factor(levels = c("0",
customerID = col_skip(),
gender = col_factor(levels = c("Female",
telco <- initial_split(telco_raw, prop = 0.8, strata = Churn)
telco_train <- training(telco)
# Since we're only running this once, we can combine testing and assessment
telco_test <- rbind(testing(telco), assessment(telco))
model_ranger <- ranger(Churn ~ ., data = telco_train)
What's the accuracy from the out-of-bag data? In other words, what is the accuracy of the model using the data that it excluded from the training of each tree?
# OOB accuracy
1 - model_ranger$prediction.error
What about the accuracy of the model on the entire training dataset?
# Training data accuracy
accuracy_vec(truth = telco_train$Churn, estimate = model_ranger$predictions)
It's the exact same! OK but what about the all-important cross-validated accuracy?
# CV accuracy
accuracy_vec(truth = telco_test$Churn, predict(model_ranger, data = telco_test)$predictions)
It's a tiny bit lower, but really close to what we saw in the training data.
If you're training a model once and not trying to tune it, you can use all of your data for training a random forest. When I'm at work, I still use CV for three reasons:
- I fiddle with tuning the model in a way that's specific to the training data.
- Often, I'm training on older data and want my model to work on data that's going to be generated in the future. I hold out the newest datapoints for testing. For example, I might train on data dated between 3 and 24 months ago and use the last two months for testing/validation. In this case, I want to account for the fact that "bias is present but its extent unknown" in holdout data.
- I like being able to compare models apples-to-apples, so I use the same holdout data and the same tests for every model I make. It's nice to be able to get the exact same metrics across all model types.