Are validation sets necessary for Random Forest Classifier?

Question

Is it necessary to have train, test and validation sets when using random forest classifier?

I understand it is important with Neural Networks but I am not understanding the importance of it with RF. I understand the idea of having a third unseen set of data to test on is important to know the model isn't overfitting, esp with Neural networks, but with RF it seems like you could almost not even have test or validation data (I know in practise this isn't true) but in theory since each tree of the forest uses a random sample (with replacement) of the training dataset.

At the moment I am missing out on approx 250 samples by keeping them unseen from the train and test set and I know the model would improve with the extra data, so is it possible to have only train and test and not designate a seperate validation set, whilst still having a reliable model?

Answer 1

is it possible to have only train and test and not designate a seperate validation set, whilst still having a reliable model?

Sure! You can train a RF on the training set, then test on the testing set. That's perfectly valid as long as the model doesn't see any of the testing data during training. (Or, better yet, you can run cross-validation since RFs are quick to train)

But if you want to tune the model's hyperparameters or do any regularization (like pruning), then you'll need a validation set. Train with the training set, use the validation set for tuning, then generate an accuracy estimate with the testing set.

Answer 2

You can use Out-of-bag error as you validation error, if you are short of data.

As you may know, Random Forest fits multiple decision trees, and for each tree it only fits on a subset of data. So data that hasn't been used for fitting a given tree is called Out of Bag data, and it could be used as your validation set 1

Sklearn in Python has a hyperparameter of Out-of-bag error

Answer 3

The existing answers are quite good but here's more detail. In the paper where he invented the random forest, Breiman touts the out-of-bag calculation as an alternative to cross-validation:

Therefore, using the out-of-bag error estimate removes the need for a set aside test set.

In the same paper, he doubles down, saying that OOB can be preferable to CV:

...unlike cross-validation, where bias is present but its extent unknown, the out-of-bag estimates are unbiased

Here's a more concrete example. Let's train a random forest based on this dataset from Kaggle.

library(conflicted)
library(tidyverse)
library(tidymodels)
library(ranger)
col_factor <- readr::col_factor
telco_raw <- read_csv(
  "archive.zip",
  col_types = cols(
    Churn = col_factor(levels = c("Yes",
                                  "No")),
    Dependents = col_factor(levels = c("Yes",
                                       "No")),
    PaperlessBilling = col_factor(levels = c("Yes",
                                             "No")),
    Partner = col_factor(levels = c("Yes",
                                    "No")),
    PhoneService = col_factor(levels = c("Yes",
                                         "No")),
    SeniorCitizen = col_factor(levels = c("0",
                                          "1")),
    customerID = col_skip(),
    gender = col_factor(levels = c("Female",
                                   "Male"))
  )
) %>% 
  na.omit()

telco <- initial_split(telco_raw, prop = 0.8, strata = Churn)
telco_train <- training(telco)
# Since we're only running this once, we can combine testing and assessment
telco_test <- rbind(testing(telco), assessment(telco))

model_ranger <- ranger(Churn ~ ., data = telco_train)

What's the accuracy from the out-of-bag data? In other words, what is the accuracy of the model using the data that it excluded from the training of each tree?

# OOB accuracy
1 - model_ranger$prediction.error
#0.7965168

Around 80%.

What about the accuracy of the model on the entire training dataset?

# Training data accuracy
accuracy_vec(truth = telco_train$Churn, estimate = model_ranger$predictions)
#0.7965168

It's the exact same! OK but what about the all-important cross-validated accuracy?

# CV accuracy
accuracy_vec(truth = telco_test$Churn, predict(model_ranger, data = telco_test)$predictions)
#0.7921708

It's a tiny bit lower, but really close to what we saw in the training data.

If you're training a model once and not trying to tune it, you can use all of your data for training a random forest. When I'm at work, I still use CV for three reasons:

1. I fiddle with tuning the model in a way that's specific to the training data.
2. Often, I'm training on older data and want my model to work on data that's going to be generated in the future. I hold out the newest datapoints for testing. For example, I might train on data dated between 3 and 24 months ago and use the last two months for testing/validation. In this case, I want to account for the fact that "bias is present but its extent unknown" in holdout data.
3. I like being able to compare models apples-to-apples, so I use the same holdout data and the same tests for every model I make. It's nice to be able to get the exact same metrics across all model types.