I'm in a situation where many models have been created, and I have their cross-validation performances as well as performance on test data. I need to select models for inclusion in a simple bagging ensemble that are most likely to generalize to new data.
Conventional wisdom would dictate choosing models with high CV performances and low correlation with each other, since each individual model should have a good chance of generalizing and there will be an error-correction effect with the model diversity.
However, it seems like given a large number of models, the odds of having a high test AUC by chance are not insignificant, and therefore choosing uncorrelated models could actually be more dangerous since that lack of correlation could indicate that they've found vastly different mechanisms to reach a high CV performance, one of which could be incorrect/overfit. Perhaps the safer way is actually to pick models within a certain range of correlation (like Pearson or Spearman in the range of 0.7-0.9, for example), to maintain some error-correction effect while ensuring that the mechanism is fairly consistent (and therefore, perhaps reliable).
I've been searching for literature on this, and haven't been able to find anything. I'd really appreciate any guidance on how to approach this, or what papers to read - thanks!