When using random forest, would using normal cross-validation and just taking the average results from multiple models with different random states give me the same results as using Repeated K-fold cross validation?

Repeated K-fold cross-validation basically repeats cross-validation with multiple different splits of the data and reports the average results.

  • $\begingroup$ Why you think that the helpfulness or not of a general technique, like repeated k-fold CV, depends on the specific ML algorithm used (RF or otherwise)? $\endgroup$ – desertnaut Mar 23 at 14:21
  • 1
    $\begingroup$ I edited the question. Basically I want to know if not doing repeated K-fold CV would still give me the same results if I just average a lot of RF models with different random states. $\endgroup$ – Artur D Mar 23 at 18:26

To the title question, yes, repeated k-fold makes sense with random forests; and to the question body, no, the results will not generally be the same as repeated model fits with one k-fold split.

The reason is that fixing one k-fold split and then repeatedly fitting random forests (with different random seeds) still only gives each forest access to $(k-1)/k$ of the data at a time. It may be easiest to think about the case when the number of trees is astronomical: the random choices for the bagging get averaged out, to the point where different random seeds don't actually matter: the forests converge to the same result, given a training split. Then the average of the forests' scores are the same for each of the splits, and so you average just $k$ scores. Compare that to repeated $k$-fold, where each of the forests converges, but are all on different training sets, and so the average happens with more variety.

Whether that has a sizable impact, or in which direction, is harder to say. Repeated $k$-fold seems like it should give more stable results, even when the number of trees is something more reasonable, because the forests are less correlated.

  • $\begingroup$ I'm not sure if I fully understand your argument. What if, with normal CV, for each of the forests we make a new split of the data into K groups, so they are not the same groups for every forest? $\endgroup$ – Artur D Mar 24 at 13:50
  • $\begingroup$ "what if...we make a new split of the data into K groups" <-- this is repeated k-fold; changing the seeds for the RF between the repeats adds some more variety, but probably not noticeable compared to the new splits. $\endgroup$ – Ben Reiniger Mar 24 at 14:01
  • $\begingroup$ So, basically, would it be enough to look at one single RF model (one seed) and do repeated k-fold? Or should I still take the average over multiple models/seeds? $\endgroup$ – Artur D Mar 24 at 14:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.