# Classification model accuracy with ensemble methods

I came through this statement in a Machine Learning text book based on law of large numbers:

Suppose you build an ensemble containing 1,000 classifiers that are individually correct only 51% of the time (barely better than random guessing). If you predict the majority voted class, you can hope for up to 75% accuracy!

I understand the analogy if we consider average over 1000 predictions but how majority votes lead to 75% accuracy from 51% (individual)?

## 1 Answer

This comes from the Binomial distribution, where you have $$n=1000$$ independent trials (models), $$p=0.51$$ of each model being right and since you care about the majority vote you want to have at least $$k=500$$ successful trials. That leads to:

$$\text{Pr}(k\geq500 \text{ models are right}) = \sum^{1000}_{k=500}\binom{1000}{k}0.51^{k}(1-0.51)^{1000-k}=0.74675\approx0.75$$

Here is how I calculated it:

    import numpy as np
from scipy.stats import binom
np.sum([binom.pmf(k,1000,0.51) for k in range(500,1001)])

• From a practical standpoint, the implicit assumption that 1000 classifiers are completely independent in making a correct classification is highly unlikely (imagine the worst case where every classifier is an instance of the same rule, in that case the ensemble would stay at 51% accuracy). This is the reason we usually try to ensemble a lot of different algorithms, hoping for maximally „independent“ classifiers. – AlexR Jul 7 '20 at 21:34