1
$\begingroup$

trying to have a better understanding of random forest algorithm here. With the same training and holdout datasets, I tried two things here:

  1. Set a small n_estimator (10), train on my training dataset and apply to my holdout dataset. If I repeat this several times, the result (e.g. correctly predicted target class) varies somewhat from run to run. My understanding is that since the # of trees is small in my model, there are variations in my model after training thus leading to different results.

  2. Set a high n_estimator (300) and do the same. Then the results don't vary. My take is that impact of high n_estimator reduces variation in the model and thus i get the same prediction every time.

So if I run my scenario 1 a bunch of times and consolidate the results (i.e. run 1 predicts A B in class 1, run 2 predicts A C in class 1, run 3 predicts D in class 1), my final results would be A B C D are in class 1. My question is: 1. Is this essentially the same as running it once with a large n_estimator? 2. Is this approach problematic because I am relying more on "guessing" (e.g. small n_estimator leads to larger variation in outcomes)?

Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

there is a parameter in estimators that is called random_state that fixes a seed for your algorithm run and reruns exactly the same so you can expect the same results at every rerun of the exact same code. It makes your code deterministic. But yes, in random forests in particular, due to their nature of averaging across all trees created when the forest is grown, variance in your result becomes less evident. So your reasoning is correct.

If you want to make your low_estimators RF produce the same results at every run, just add random_state and give it a random number

Improve this answer
$\endgroup$
5
  • $\begingroup$ By setting random_state = None, I am introducing some randomness (stochasticity) into the model. Is there generally considered a bad thing? If i have a routine process with new data coming in say every week and I want to re-run the model, is it generally better / recommended to set a random_state? $\endgroup$
    – adjfac
    Oct 3, 2019 at 20:36
  • 1
    $\begingroup$ Yes, in a general fashion, it's not a good thing to just pick the random_state with the highest accuracy or lowest error rate. But in the validation phase of a model, it's better to fix the seed, and tune the hyper-parameters , for example , and have a good estimate what's a good parameter and whats not. If you keep the randomness involved, you won't be sure if that parameter is being a good element or bad one for your model. $\endgroup$
    – Blenz
    Oct 3, 2019 at 21:27
  • 1
    $\begingroup$ You could always set a random_state for your whole process of feature engineering/modelling/validation, and then iterate over multiple random states and average your results $\endgroup$
    – Blenz
    Oct 3, 2019 at 21:28
  • 1
    $\begingroup$ And for your case specifically, it would be good if you set a random_state, a low number of estimators and have a general idea on how your model performs, because the variation in the result due to not setting a random state up isn't that big. You save the computational power for having extra trees grown, and spend less time validating your model. $\endgroup$
    – Blenz
    Oct 3, 2019 at 21:32
  • 1
    $\begingroup$ And one last note, a significant difference in the error rate due to random_state changes indicates that your model isn't stable, let alone ready to be put in production to encounter unseen data. You can test different random states for your model after you're done to evaluate its stability. $\endgroup$
    – Blenz
    Oct 3, 2019 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.