Selecting models for ensemble from large group of models with high uncertaintly

Question

I'm in a situation where many models have been created, and I have their cross-validation performances as well as performance on test data. I need to select models for inclusion in a simple bagging ensemble that are most likely to generalize to new data.

Conventional wisdom would dictate choosing models with high CV performances and low correlation with each other, since each individual model should have a good chance of generalizing and there will be an error-correction effect with the model diversity.

However, it seems like given a large number of models, the odds of having a high test AUC by chance are not insignificant, and therefore choosing uncorrelated models could actually be more dangerous since that lack of correlation could indicate that they've found vastly different mechanisms to reach a high CV performance, one of which could be incorrect/overfit. Perhaps the safer way is actually to pick models within a certain range of correlation (like Pearson or Spearman in the range of 0.7-0.9, for example), to maintain some error-correction effect while ensuring that the mechanism is fairly consistent (and therefore, perhaps reliable).

I've been searching for literature on this, and haven't been able to find anything. I'd really appreciate any guidance on how to approach this, or what papers to read - thanks!

Answer 1

The general principle of ensemble learning is indeed to rely on the diversity of the individual learners rather than their performance. Therefore it's ok to include models which perform well by chance (typically because they're overfit), since if there is sufficient diversity across the models it's very unlikely that two models would be overfit in the same way, i.e. that they would wrongly predict the same instances by chance. So if most models perform reasonably well in general through different mechanisms, it is expected that for any given instance most models will predict the right answer and only a few will be wrong.

The risk of using models which have a quite high correlation together is to defeat the purpose of ensemble learning and obtain a performance similar to the best individual learner.

Note also that the risk of overfitting is quite low if the individual learners have been tested with CV. It could also be worth checking the variance of the performance across CV runs (high variance indicating potential instability), but even that is not so useful in my experience: it's really the diversity of the learners which makes ensemble learning work optimally.

But as usual a lot depends on the actual data/task, so I'd suggest properly testing a few different approaches.

Answer 2

You can treat model ensembling as a hyperparameter, then use cross validation to compare different combinations of models. This strategy transforms it to an empirical problem and finds the best solution for your specific use case.