1. I want to avoid overfitting in random forest. In this regard, I intend to use mtry, nodesize, and maxnodes etc. Could you please help me choose values for these parameters? I am using R.

  2. Also, if possible, please tell me how I can use k-fold cross validation for random forest (in R).


4 Answers 4


Relative to other models, Random Forests are less likely to overfit but it is still something that you want to make an explicit effort to avoid. Tuning model parameters is definitely one element of avoiding overfitting but it isn't the only one. In fact I would say that your training features are more likely to lead to overfitting than model parameters, especially with a Random Forests. So I think the key is really having a reliable method to evaluate your model to check for overfitting more than anything else, which brings us to your second question.

As alluded to above, running cross validation will allow to you avoid overfitting. Choosing your best model based on CV results will lead to a model that hasn't overfit, which isn't necessarily the case for something like out of the bag error. The easiest way to run CV in R is with the caret package. A simple example is below:

> library(caret)
> data(iris)
> tr <- trainControl(method = "cv", number = 5)
> train(Species ~ .,data=iris,method="rf",trControl= tr)
Random Forest 

150 samples
  4 predictor
  3 classes: 'setosa', 'versicolor', 'virginica' 

No pre-processing
Resampling: Cross-Validated (5 fold) 

Summary of sample sizes: 120, 120, 120, 120, 120 

Resampling results across tuning parameters:

  mtry  Accuracy  Kappa  Accuracy SD  Kappa SD  
  2     0.96      0.94   0.04346135   0.06519202
  3     0.96      0.94   0.04346135   0.06519202
  4     0.96      0.94   0.04346135   0.06519202

Accuracy was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 2. 
  • $\begingroup$ Thank you very much for the reply and advice. Just want to know how different is using a caret package for feature selection when compared to selecting important variables using the randomForest algorithm? $\endgroup$
    – Arun
    Commented Aug 5, 2015 at 10:39
  • $\begingroup$ That sounds like a different question to post -- the differences are vast given that you run things like recursive feature selection using any algorithm of your choice in caret. $\endgroup$
    – David
    Commented Aug 5, 2015 at 14:38
  • $\begingroup$ @David What do you mean by "which isn't necessarily the case for something like out of the bag error"? Do you mean that out of the bag estimation is not a good way to avoid overfitting? $\endgroup$
    – Pierre
    Commented Apr 13, 2018 at 9:43

@xof6 is correct in the sense that the more depth the model has the more it tends to overfit, but I wanted to add some more parameters that might be useful to you. I do not know which package you are using with R and I am not familiar with R at all, but I think there must be counterparts of these parameters implemented there.

Number of trees - The bigger this number, the less likely the forest is to overfit. This means that as each decision tree is learning some aspect of the training data, you are getting more options to choose from, so to speak. Number of features - This number constitutes how many features each individual tree learns. As this number grows, the trees get more and more complicated, hence they are learning patterns that might not be there in the test data. It will take some experimenting to find the right value, but such is machine learning. Experiment with the general depth as well, as we mentioned!


Here is a nice link on that on stackexchange, however my general experience is the more depth the model has the more it tends to overfit.

  • $\begingroup$ This makes sense with a single tree classifier. However, I have read on these forums that random forests are protected from over-fitting and that trees should be grown fully (i.e. nodesize = 1 for classification). $\endgroup$ Commented Nov 23, 2017 at 11:25

I always, decrease mtry until error on train dataset increases, then I lower nodesize and depth until difference between error on train and dataset stops to decrease


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.