i'm trying to find the best n_estimator value on a Random Forest ML model by running this loop:

for i in r:
   RF_model_i = RandomForestClassifier(criterion="gini",   n_estimators=i, oob_score=True)
   RF_model_i.id = [i]  # dynamically add fields to objects
   RF_model_i.fit(X_train, y_train)
   y_predict_i =   RF_model_i.predict(X_test)
   accuracy_i = [accuracy_score(y_test, y_predict_i), i]
   results.append(accuracy_i) # put the result on a list within the for-loop
  1. question #1 ** What i like to understand is if this could be a good way to understand how to decide the n_estimator parameter and possibly why (i'm not so sure it is) 2.**question #2 If what i'm doing have some sense, could be a good idea extend this loop to all of other main parameters?

What i obtain is this: a level of accuracy for a number of estimator associated with which i runned the random forest algorithm

enter image description here Thank you


The number of trees in a random forest doesn't really need to be tuned, at least not in the same way as other hyperparameters. Adding more trees just stabilizes the results (you're averaging more samples from a distribution of trees); you want enough trees to get stable results, and adding more won't hurt except for computational resources.

More directly (question 1), you could instead train the RF with, say, 1000 n_estimators, then just grab the predictions from each tree and average only the first 100 of them, the first 200, etc. Since the trees are built independently, this is essentially the same as training repeatedly on 100, then 200, etc. trees from scratch (but of course faster).

More generally (question 2), yes, this is one-dimensional grid search. Hyperparameters may be interdependent though, so doing independent 1D searches will probably be suboptimal. Look into full grid search, random search, and maybe more advanced hyperparameter optimization methods.

  • $\begingroup$ thank you @Ben, which kind of more advanced hyper parameter optimization methods do you suggest me to look into? i'm quite beginner on this so i'd like to understadn what to look for . $\endgroup$ – pellerossa pelles Mar 19 '20 at 14:59
  • $\begingroup$ For random forest, I'd stick with grid/random searches. If you have time/desire to explore (but I wouldn't count on much improvement here), you could try one of various Bayesian optimization methods (hyperopt, scitkit-opt), or multi-armed bandits. datascience.stackexchange.com/questions/tagged/… $\endgroup$ – Ben Reiniger Mar 19 '20 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.