Why are ensembles so unreasonably effective

Question

It seems to have become axiomatic that an ensemble of learners leads to the best possible model results - and it is becoming far rarer, for example, for single models to win competitions such as Kaggle. Is there a theoretical explanation for why ensembles are so darn effective?

Answer 1

For a specific model you feed it data, choose the features, choose hyperparameters etcetera. Compared to the reality it makes a three types of mistakes:

• Bias (due to too low model complexity, a sampling bias in your data)
• Variance (due to noise in your data, overfitting of your data)
• Randomness of the reality you are trying to predict (or lack of predictive features in your dataset)

Ensembles average out a number of these models. The bias due to sampling bias will not be fixed for obvious reasons, it can fix some of the model complexity bias, however the variance mistakes that are made are very different over your different models. Especially low correlated models make very different mistakes in this areas, certain models perform well in certain parts of your feature space. By averaging out these models you reduce this variance quite a bit. This is why ensembles shine.

Answer 2

The selected answer is fantastic, but I would like to add two things:

1. It has been observed that averaging human predictions gives better predictions than any individual prediction. This is known as the wisdom of the crowd. Now, you could argue that it is because some people have different information, so you are effectively averaging information. But no, this is true even for tasks such as guessing the number of beans in a jar. I hypothesize it has to do with some of the reasons given above about data mining models.
2. Some techniques such as the dropout method in neural networks (where in each iteration during training you use only a chunk of your neural network) give results similar to an ensemble of neural networks. The rationale is that you are effectively forcing nodes to do the same predictor work as the other nodes, effectively creating a meta-ensemble. I am saying this to make the point that we may be able to introduce some of the advantages of ensembles in traditional models.

Answer 3

Ensembles win at prediction for theoretical and practical reasons.

There is a fundamental theory of optimal forecasting, if we mean predicting the next event in a sequence based on knowledge of previous events. Solomonoff prediction (Solomonoff 1964) is provably optimal in several senses, including that it “will learn to correctly predict any computable sequence with only the absolute minimum amount of data.” (Hutter, Legg & Vitanyi 2007) A Solomonoff predictor weights all programs compatible with the existing data, according to the Kolmogorov complexity of the program and the probability that program assigns to the data so far, combining Epicurean (“keep all theories”) and Ockham (“prefer simple theories”) philosophies in a Bayesian framework.

The optimality properties of Solomonoff prediction explain the robust finding you refer to: averaging over models, sources, or experts improves predictions, and averaged predictions outperform even the best single predictor. The various ensemble methods seen in practice can be seen as computable approximations to Solomonoff prediction -- and some like MML (Wallace 2005) explicitly explore the ties, though most do not.

Wallace (2005) notes that a Solomonoff predictor is not parsimonious – it keeps an infinite pool of models -- but most of the predictive power inevitably falls to a relatively small set of models. In some domains the single best model (or family of nearly indistinguishable models) may account for a large portion of the predictive power and outperform generic ensembles, but in complex domains with little theory most likely no single family captures the majority of the posterior probability, and therefore averaging over the plausible candidates should improve predictions. To win the Netflix prize, the Bellkor team blended over 450 models (Koren 2009).

Humans typically seek a single good explanation: in "high-theory" domains like physics, these work well. Indeed if they capture the underlying causal dynamics, they should be nearly unbeatable. But where the available theories do not closely fit the phenomena (say, film recommendation or geopolitics), single models will underperform: all are incomplete, so none should dominate. Thus the recent emphasis on ensembles (for machine learning) and Wisdom of the Crowds (for experts), and the success of programs like IARPA ACE and specifically the Good Judgment Project (Tetlock & Gardiner 2015).

References

• M. Hutter, S. Legg, and P. Vitanyi, “Algorithmic probability,” Scholarpedia, vol. 2, 2007, p. 2572.
• Y. Koren, “The BellKor Solution to the Netflix Grand Prize,” 2009.
• Solomonoff, Ray (March 1964). "A Formal Theory of Inductive Inference Part I" (PDF). Information and Control 7 (1): 1–22. doi:10.1016/S0019-9958(64)90223-2.
• Solomonoff, Ray (June 1964). "A Formal Theory of Inductive Inference Part II" (PDF). Information and Control 7 (2): 224–254. doi:10.1016/S0019-9958(64)90131-7.
• P.E. Tetlock, Expert Political Judgment: How Good Is It? How Can We Know?, Princeton University Press, 2005.
• Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. New York: Crown.
• C.S. Wallace, Statistical and Inductive Inference by Minimum Message Length, Springer-Verlag, 2005.