4
$\begingroup$

I came through this statement in a Machine Learning text book based on law of large numbers:

Suppose you build an ensemble containing 1,000 classifiers that are individually correct only 51% of the time (barely better than random guessing). If you predict the majority voted class, you can hope for up to 75% accuracy!

I understand the analogy if we consider average over 1000 predictions but how majority votes lead to 75% accuracy from 51% (individual)?

$\endgroup$
2
$\begingroup$

This comes from the Binomial distribution, where you have $n=1000$ independent trials (models), $p=0.51$ of each model being right and since you care about the majority vote you want to have at least $k=500$ successful trials. That leads to:

$$\text{Pr}(k\geq500 \text{ models are right}) = \sum^{1000}_{k=500}\binom{1000}{k}0.51^{k}(1-0.51)^{1000-k}=0.74675\approx0.75$$

Here is how I calculated it:

    import numpy as np
    from scipy.stats import binom
    np.sum([binom.pmf(k,1000,0.51) for k in range(500,1001)])
$\endgroup$
1
  • 1
    $\begingroup$ From a practical standpoint, the implicit assumption that 1000 classifiers are completely independent in making a correct classification is highly unlikely (imagine the worst case where every classifier is an instance of the same rule, in that case the ensemble would stay at 51% accuracy). This is the reason we usually try to ensemble a lot of different algorithms, hoping for maximally „independent“ classifiers. $\endgroup$ – AlexR Jul 7 '20 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.