Is it necessary to have train, test and validation sets when using random forest classifier?

I understand it is important with Neural Networks but I am not understanding the importance of it with RF. I understand the idea of having a third unseen set of data to test on is important to know the model isn't overfitting, esp with Neural networks, but with RF it seems like you could almost not even have test or validation data (I know in practise this isn't true) but in theory since each tree of the forest uses a random sample (with replacement) of the training dataset.

At the moment I am missing out on approx 250 samples by keeping them unseen from the train and test set and I know the model would improve with the extra data, so is it possible to have only train and test and not designate a seperate validation set, whilst still having a reliable model?


is it possible to have only train and test and not designate a seperate validation set, whilst still having a reliable model?

Sure! You can train a RF on the training set, then test on the testing set. That's perfectly valid as long as the model doesn't see any of the testing data during training. (Or, better yet, you can run cross-validation since RFs are quick to train)

But if you want to tune the model's hyperparameters or do any regularization (like pruning), then you'll need a validation set. Train with the training set, use the validation set for tuning, then generate an accuracy estimate with the testing set.

  • $\begingroup$ Thank you for the answer! That makes things clearer. I want to just clarify (and apologies if this is really stupid) I have done parameter tuning so I know what my best parameters are, now when I run my model and I have the best params already set, that means I do not need to use a validation set anymore? I only needed to use it once to find out the ideal parameters yes? $\endgroup$ – codiearcher Oct 8 '19 at 13:39
  • 1
    $\begingroup$ By "run your model", do you mean running the model in production? If so, then you're absolutely right. Your parameters were set during training, and now you hold them constant. If you're still talking about testing: As long as the testing set was not seen during the training/tuning process, then you're in the clear! $\endgroup$ – zachdj Oct 8 '19 at 14:03

You can use Out-of-bag error as you validation error, if you are short of data.

As you may know, Random Forest fits multiple decision trees, and for each tree it only fits on a subset of data. So data that hasn't been used for fitting a given tree is called Out of Bag data, and it could be used as your validation set 1

Sklearn in Python has a hyperparameter of Out-of-bag error


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.