3
$\begingroup$

Is it necessary to have train, test and validation sets when using random forest classifier?

I understand it is important with Neural Networks but I am not understanding the importance of it with RF. I understand the idea of having a third unseen set of data to test on is important to know the model isn't overfitting, esp with Neural networks, but with RF it seems like you could almost not even have test or validation data (I know in practise this isn't true) but in theory since each tree of the forest uses a random sample (with replacement) of the training dataset.

At the moment I am missing out on approx 250 samples by keeping them unseen from the train and test set and I know the model would improve with the extra data, so is it possible to have only train and test and not designate a seperate validation set, whilst still having a reliable model?

$\endgroup$
3
$\begingroup$

is it possible to have only train and test and not designate a seperate validation set, whilst still having a reliable model?

Sure! You can train a RF on the training set, then test on the testing set. That's perfectly valid as long as the model doesn't see any of the testing data during training. (Or, better yet, you can run cross-validation since RFs are quick to train)

But if you want to tune the model's hyperparameters or do any regularization (like pruning), then you'll need a validation set. Train with the training set, use the validation set for tuning, then generate an accuracy estimate with the testing set.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer! That makes things clearer. I want to just clarify (and apologies if this is really stupid) I have done parameter tuning so I know what my best parameters are, now when I run my model and I have the best params already set, that means I do not need to use a validation set anymore? I only needed to use it once to find out the ideal parameters yes? $\endgroup$ Oct 8 '19 at 13:39
  • 1
    $\begingroup$ By "run your model", do you mean running the model in production? If so, then you're absolutely right. Your parameters were set during training, and now you hold them constant. If you're still talking about testing: As long as the testing set was not seen during the training/tuning process, then you're in the clear! $\endgroup$
    – zachdj
    Oct 8 '19 at 14:03
4
$\begingroup$

You can use Out-of-bag error as you validation error, if you are short of data.

As you may know, Random Forest fits multiple decision trees, and for each tree it only fits on a subset of data. So data that hasn't been used for fitting a given tree is called Out of Bag data, and it could be used as your validation set 1

Sklearn in Python has a hyperparameter of Out-of-bag error

$\endgroup$
1
  • 1
    $\begingroup$ In that case, you should assign Bootstrap = True arguement if you need OOB error. $\endgroup$
    – Mari
    Apr 7 '20 at 8:34
1
$\begingroup$

The existing answers are quite good but here's more detail. In the paper where he invented the random forest, Breiman touts the out-of-bag calculation as an alternative to cross-validation:

Therefore, using the out-of-bag error estimate removes the need for a set aside test set.

In the same paper, he doubles down, saying that OOB can be preferable to CV:

...unlike cross-validation, where bias is present but its extent unknown, the out-of-bag estimates are unbiased

Here's a more concrete example. Let's train a random forest based on this dataset from Kaggle.

library(conflicted)
library(tidyverse)
library(tidymodels)
library(ranger)
col_factor <- readr::col_factor
telco_raw <- read_csv(
  "archive.zip",
  col_types = cols(
    Churn = col_factor(levels = c("Yes",
                                  "No")),
    Dependents = col_factor(levels = c("Yes",
                                       "No")),
    PaperlessBilling = col_factor(levels = c("Yes",
                                             "No")),
    Partner = col_factor(levels = c("Yes",
                                    "No")),
    PhoneService = col_factor(levels = c("Yes",
                                         "No")),
    SeniorCitizen = col_factor(levels = c("0",
                                          "1")),
    customerID = col_skip(),
    gender = col_factor(levels = c("Female",
                                   "Male"))
  )
) %>% 
  na.omit()

telco <- initial_split(telco_raw, prop = 0.8, strata = Churn)
telco_train <- training(telco)
# Since we're only running this once, we can combine testing and assessment
telco_test <- rbind(testing(telco), assessment(telco))

model_ranger <- ranger(Churn ~ ., data = telco_train)

What's the accuracy from the out-of-bag data? In other words, what is the accuracy of the model using the data that it excluded from the training of each tree?

# OOB accuracy
1 - model_ranger$prediction.error
#0.7965168

Around 80%.

What about the accuracy of the model on the entire training dataset?

# Training data accuracy
accuracy_vec(truth = telco_train$Churn, estimate = model_ranger$predictions)
#0.7965168

It's the exact same! OK but what about the all-important cross-validated accuracy?

# CV accuracy
accuracy_vec(truth = telco_test$Churn, predict(model_ranger, data = telco_test)$predictions)
#0.7921708

It's a tiny bit lower, but really close to what we saw in the training data.

If you're training a model once and not trying to tune it, you can use all of your data for training a random forest. When I'm at work, I still use CV for three reasons:

  1. I fiddle with tuning the model in a way that's specific to the training data.
  2. Often, I'm training on older data and want my model to work on data that's going to be generated in the future. I hold out the newest datapoints for testing. For example, I might train on data dated between 3 and 24 months ago and use the last two months for testing/validation. In this case, I want to account for the fact that "bias is present but its extent unknown" in holdout data.
  3. I like being able to compare models apples-to-apples, so I use the same holdout data and the same tests for every model I make. It's nice to be able to get the exact same metrics across all model types.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.