Find the optimal n_estimator by looping the model accuracy indicator in random forest algorithm - python

Question

i'm trying to find the best n_estimator value on a Random Forest ML model by running this loop:

for i in r:
   RF_model_i = RandomForestClassifier(criterion="gini",   n_estimators=i, oob_score=True)
   RF_model_i.id = [i]  # dynamically add fields to objects
   RF_model_i.fit(X_train, y_train)
   y_predict_i =   RF_model_i.predict(X_test)
   accuracy_i = [accuracy_score(y_test, y_predict_i), i]
   results.append(accuracy_i) # put the result on a list within the for-loop

1. question #1 ** What i like to understand is if this could be a good way to understand how to decide the n_estimator parameter and possibly why (i'm not so sure it is) 2.**question #2 If what i'm doing have some sense, could be a good idea extend this loop to all of other main parameters?

What i obtain is this: a level of accuracy for a number of estimator associated with which i runned the random forest algorithm

Thank you

Answer 1

The number of trees in a random forest doesn't really need to be tuned, at least not in the same way as other hyperparameters. Adding more trees just stabilizes the results (you're averaging more samples from a distribution of trees); you want enough trees to get stable results, and adding more won't hurt except for computational resources.

More directly (question 1), you could instead train the RF with, say, 1000 n_estimators, then just grab the predictions from each tree and average only the first 100 of them, the first 200, etc. Since the trees are built independently, this is essentially the same as training repeatedly on 100, then 200, etc. trees from scratch (but of course faster).

More generally (question 2), yes, this is one-dimensional grid search. Hyperparameters may be interdependent though, so doing independent 1D searches will probably be suboptimal. Look into full grid search, random search, and maybe more advanced hyperparameter optimization methods.