1
$\begingroup$

I noticed that I am getting different feature importance results with each random forest run even though they are using the same parameters. Now, I know that a random forest model takes observations randomly which is causing the importance levels to vary. This is especially shown for the less important variables.

My question is how does one interpret the variance in random forest results when running it multiple times? I know that one can reduce the instability level of results by increasing the number of trees; however, this doesn't really tell me if my feature importance results are "true" though they may be true for that specific run (but not necessarily for a separate run).

Even if I were to take an extremely large number of trees and average the feature importance results for each variable, that still doesn't necessarily confirm that it will produce the same importance results if I repeat that exact same process again.

Additionally, I have tried it with an extremely large number of trees and still got a slight variation (it did significantly reduce the variance of my results) in my feature importance results between runs.

Is there any method that I can use to interpret this variance of importance between runs?

I cannot set a seed because I need stable (similar) results across different seeds.

Any help at all would be greatly appreciated!

$\endgroup$
2
  • $\begingroup$ Can you explain 'I cannot set a seed because I need stable (similar) results across different seeds.' starting with different seeds is often the main reason results can change $\endgroup$ May 13, 2022 at 17:15
  • $\begingroup$ @RalphWinters Forgive my ignorance, how does one know the true variable importance if the importance ranking is not stable when run across different seeds? $\endgroup$
    – Detr4
    May 13, 2022 at 17:23

1 Answer 1

2
$\begingroup$

Random Forests are full of 'randomness', from selecting and resampling the actual data (bootstrapping) to selection of the best features that go into the individual decision trees. So with all of this sampling going on the starting seed will affect all of these intermediate results as well as the final set of trees. Since you asked about the feature importance it will also affect the ranking as well. So it is always best to keep the seed the same.

If you results are changing, and you are doing multiple runs, averaging the feature importance of all of the runs should give you a good idea of what the 'true' value should be.

$\endgroup$
5
  • $\begingroup$ Thank you for the explanation and suggestion. So, in essence, it is not possible to produce truly consistent importance rankings with varying seeds? I am curious if journals would accept averaging the results over multiple seeds. Do you have any alternative model suggestions that are perhaps similar to random forests? Perhaps I would be better off with a deterministic modeling apporach? $\endgroup$
    – Detr4
    May 13, 2022 at 19:39
  • $\begingroup$ The purpose of the seed is really for reproducibility. Changing the seed shouldn't change the output all that much if run over a large number of bootstraps. However it is prudent to check results against other algorithms. I am thinking you should try other techniques such as lasso regression, or SVM and see if you come up with similar results. $\endgroup$ May 14, 2022 at 17:14
  • $\begingroup$ Thank you for the clarification! I will be sure to try these techniques. Also, would it be theoretically possible to use such a large number of trees that your importance rankings become 'static' even when run on separate seeds? My data contains 19705 observations and 18 variables. 11 variables heavily vary from run to run on 200 trees and only 4 of them vary when using 2000 trees. $\endgroup$
    – Detr4
    May 14, 2022 at 17:32
  • $\begingroup$ As you increase the number of trees that will converge upon the 'true' value (given your hyper-parameters), only because the assumption is that you are increasing the n of the 'sampling distribution'. also I would not refer to it as static, for given another set of data similar to yours, it is possible to come up with different results. Remember, Feature importance is measure by accuracy in the model and not by any cause and effect. Also note the feature importance in RF has been criticized in the past and it is hard to determine what their statistical properties really are. $\endgroup$ May 14, 2022 at 18:37
  • $\begingroup$ Masterful, thank you! $\endgroup$
    – Detr4
    May 14, 2022 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.