Does increasing the n_estimators parameter in decision trees always increase accuracy

Question

I'm using some ML algorithms (from sklearn lib) and on most of them there is a parameter n_estimators which is (if I understood well) the number of used trees. Intuitevely I would say that the more trees you use, the more accurate results you get. It turned out to be not exactly true, sometimes, a very few number of trees give much better results, but I can't figure out why ?

Edit

Some precisions: this is a regression problem, with a dataset containing about 4000 samples and 500 features. I'm using GradientBoostingRegressor, ExtraTreeRegressor, AdaBoost, RandomForest.

Edit 2

Additional info: I use a cross-validation with KFold=10 to evaluate the accuracy of the algorithms. The best n_estimators value seems to be 50, which give a R2 score of ~56/57% +- 8% for all above cited algo. When I try to increase it, the score quickly decreases. I tried several values, from 100 to 500, it keeps decreasing even reaching 52%.

Answer 1

There are a lot of misconceptions about regression random forest. Those misconceptions about regression rf are seen also in classification rf, but are less visible. The one I will present here is that regression random forests do not overfit. Well, this is not true.

Studying the statistical properties of the random forests shows that the bootstrapping procedure decreases the variance and maintain the bias. This property should be understood under the bias-variance tradeoff framework. It is clear that the random forests approximate an expectation, which means the mean of the true structure remains the same, while the variance is reduced. From this perspective, the random forests do not overfit. There is a problem, however, that problem is the sample itself which is used for training. The expectation is taken conditional on data. And if the data is not representative of the problem, it is normal that in the limit when the number of the tree grows to infinity. In plain English, this means that the regression forest will learn too well the data and if the data is not representative, then the results are bad. In which way the data might not be representative? In many ways, one would be that you do not have enough data points in all region of interests for example. This problem is seen often with testing error, so it might not affect you so much, but is possible to see that in CV also.

Another issue with regression trees is the number of significant variables and the number of nonsignificant variables in your data set. It is known that when you have few interesting input variables and a large number of noise variables the regression forests does not behave well. Boosting procedures does not have this behavior. There is a good reason for that. Regression forests produce more uninteresting trees which have the potential to move the learned structure away from the true underlying structure. For boosting this does not happen since at each iteration only the region of interests have large weight, so the already learned regions are affected less. The remedy would be to play with the number of variables selected on learning time.

There is a drawback however even if you increase the number of variables take into account at learning time. Consider two randomly grown trees. If you would have 100 input variables and select 10 of them for learning, there are small chances that the trees look similar. If instead you would select 50 variables for learning then your trees have better chances to look similar. This is translated into the fact that if you increase the number of variables for testing candidates at learning time, then the correlations between them increases. If the correlation increases, then they will not be able to learn a more complex structure, because of their correlations. If the number of variables selected is small you have the potential to learn more due to diversity, but if the significant variables are small compared to nonsignificant, this would lead to learn noise, too much noise. This affects most of the time the CV performance.