Does modeling with Random Forests require cross-validation?

Question

As far as I've seen, opinions tend to differ about this. Best practice would certainly dictate using cross-validation (especially if comparing RFs with other algorithms on the same dataset). On the other hand, the original source states that the fact OOB error is calculated during model training is enough of an indicator of test set performance. Even Trevor Hastie, in a relatively recent talks says that "Random Forests provide free cross-validation". Intuitively, this makes sense to me, if training and trying to improve a RF-based model on one dataset.

Can someone please lay out the arguments for and against the need for cross-validation with random forests?

Answer 1

By default random forest picks up 2/3rd data for training and rest for testing for regression and almost 70% data for training and rest for testing during classification.By principle since it randomizes the variable selection during each tree split it's not prone to overfit unlike other models.However if you want to use CV using nfolds in sklearn you can still use the concept of hold out set such as oob_score(out of bag)=True which shows model performance with or without using CV. So in a nutshell using oob_score=True with or without nfolds can itself tell whether using CV is good for your data.Generally if your target is following a certain distribution and you don't have much observation data with you then using CV will not give much improvement.

Answer 2

One key difference is that cross validation ensures all samples will appear in the training and test sets, so 100% of your data gets used at some point for training and for testing.

Depending on the size of your dataset the bootstrapping , sampling with replacement, occurring in the random forest will not guarantee the splits the trees see will contain all instances. If you have enough trees in your forest the OOB estimate should asymptotically converge towards the best OOB estimate value.

The accuracy for both methods will to some degree be data dependent so it may be prudent to compare both methods on the particular data you have in front of you and see if CV and RF OOB estimates give similar values.

If they do not, then it would be worth exploring further estimates of the true error rate, perhaps by much higher vales of K in CV.

Answer 3

I did some tests on a data set of 50k rows, using sklearn.RandomForestRegressor.

I get significantly different scores - I'm using a normalized gini for the metric - depending on whether I use rf.oob_prediction_ (0.2927) or a KFold CV (0.3258 for 7 folds and 0.3236 for 3 folds).

With that, it appears your point about "especially if comparing RFs with other algorithms on the same dataset" is a strong consideration towards using manual CV rather than relying on the OOB prediction.