30
$\begingroup$

I have been reading around about Random Forests but I cannot really find a definitive answer about the problem of overfitting. According to the original paper of Breiman, they should not overfit when increasing the number of trees in the forest, but it seems that there is not consensus about this. This is creating me quite some confusion about the issue.

Maybe someone more expert than me can give me a more concrete answer or point me in the right direction to better understand the problem.

$\endgroup$
  • 3
    $\begingroup$ All algorithms will overfit to some degree. It's not about picking something that doesn't overfit, it's about carefully considering the amount of overfitting and the form of the problem you're solving to maximize more relevant metrics. $\endgroup$ – indico Aug 23 '14 at 18:16
  • 1
    $\begingroup$ ISTR that Breiman had a proof based on the Law of Large Numbers. Has someone discovered a flaw in that proof? $\endgroup$ – JenSCDC Aug 28 '14 at 1:18
  • $\begingroup$ @AndyBlankertz ISTR = internetslang.com/ISTR-meaning-definition.asp ? $\endgroup$ – Hack-R Nov 3 '15 at 3:15
26
$\begingroup$

Every ML algorithm with high complexity can overfit. However, the OP is asking whether an RF will not overfit when increasing the number of trees in the forest.

In general, ensemble methods reduces the prediction variance to almost nothing, improving the accuracy of the ensemble. If we define the variance of the expected generalization error of an individual randomized model as:

From here, the variance of the expected generalization error of an ensemble corresponds to:

where p(x) is the Pearson’s correlation coefficient between the predictions of two randomized models trained on the same data from two independent seeds. If we increase the number of DT's in the RF, larger M, the variance of the ensemble decreases when ρ(x)<1. Therefore, the variance of an ensemble is strictly smaller than the variance of an individual model.

In a nutshell, increasing the number of individual randomized models in an ensemble will never increase the generalization error.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ That's definitely what Leo Breiman and the theory says, but empirically it seems like they definitely do overfit. For example I currently have a model with 10-fold CV MSE of 0.02 but when measured against the ground truth the CV MSE is .4. OTOH if I reduce tree depth and tree number the model performance improves significantly. $\endgroup$ – Hack-R Feb 18 '16 at 14:41
  • 5
    $\begingroup$ If you reduce the tree depth is a different case because you are adding regularisation, which will decrease the overfitting. Try to plot the MSE when you increase the number of trees while keeping the rest of parameters unchanged. So, you have MSE in the y-axis and num_tress in the x-axis. You will see that when adding more trees, the error decreases fast, and then it has a plateau; but it will never increase. $\endgroup$ – tashuhka Feb 19 '16 at 13:43
12
$\begingroup$

You may want to check cross-validated - a stachexchange website for many things, including machine learning.

In particular, this question (with exactly same title) has already been answered multiple times. Check these links.

But I may give you the short answer to it: yes, it does overfit, and sometimes you need to control the complexity of the trees in your forest, or even prune when they grow too much - but this depends on the library you use for building the forest. E.g. in randomForest in R you can only control the complexity

| improve this answer | |
$\endgroup$
9
$\begingroup$
  1. The Random Forest does overfit.
  2. The Random Forest does not increase generalization error when more trees are added to the model. The generalization variance is going to zero with more trees used.

I've made a very simple experiment. I have generated the synthetic data:

y = 10 * x + noise

I've train two Random Forest models:

  • one with full trees
  • one with pruned trees

The model with full trees has lower train error but higher test error than the model with pruned trees. The responses of both models:

responses

It is clear evidence of overfitting. Then I took the hyper-parameters of the overfitted model and check the error while adding at each step 1 tree. I got the following plot:

growing trees

As you can see the overfit error is not changing when adding more trees but the model is overfitted. The experiment with code examples is described in my blog post.

| improve this answer | |
$\endgroup$
  • $\begingroup$ for completeness, could you explain how did you prune the trees ? $\endgroup$ – Thomas Aug 31 at 8:18
  • $\begingroup$ I've set min_samples_leaf=25 in RF (the default is 1). Similar can be obtained with setting max_depth for example with 6. $\endgroup$ – pplonski Aug 31 at 9:45
  • $\begingroup$ perfect thanks ! $\endgroup$ – Thomas Aug 31 at 12:30
1
$\begingroup$

STRUCTURED DATASET -> MISLEADING OOB ERRORS

I've found interesting case of RF overfitting in my work practice. When data are structured RF overfits on OOB observations.

Detail :

I try to predict electricity prices on electricity spot market for each single hour (each row of dataset contain price and system parameters (load, capacities etc.) for that single hour).
Electricity prices are created in batches (24 prices created on electricity market in one fixing in one moment of time).
So OOB obs for each tree are random subsets of set of hours, but if you predict next 24 hours you do it all at once (in first moment you obtain all system parameters, then you predict 24 prices, then there is an fixing which produces those prices), so its easier to make OOB predictions, then for the whole next day. OOB obs are not contained in 24-hour blocks, but dispersed uniformly, as there is an autocorrelation of prediction errors its easier to predict price for single hour which is missing then for whole block of missing hours.

easier to predict in case of error autocorrelation :
known, known, prediction, known, prediction - OBB case
harder one :
known, known, known, prediction, prediction - real world prediction case

I hope its interesting

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.