4
$\begingroup$

I came through this statement in a Machine Learning text book based on law of large numbers:

Suppose you build an ensemble containing 1,000 classifiers that are individually correct only 51% of the time (barely better than random guessing). If you predict the majority voted class, you can hope for up to 75% accuracy!

I understand the analogy if we consider average over 1000 predictions but how majority votes lead to 75% accuracy from 51% (individual)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

This comes from the Binomial distribution, where you have $n=1000$ independent trials (models), $p=0.51$ of each model being right and since you care about the majority vote you want to have at least $k=500$ successful trials. That leads to:

$$\text{Pr}(k\geq500 \text{ models are right}) = \sum^{1000}_{k=500}\binom{1000}{k}0.51^{k}(1-0.51)^{1000-k}=0.74675\approx0.75$$

Here is how I calculated it:

    import numpy as np
    from scipy.stats import binom
    np.sum([binom.pmf(k,1000,0.51) for k in range(500,1001)])
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ From a practical standpoint, the implicit assumption that 1000 classifiers are completely independent in making a correct classification is highly unlikely (imagine the worst case where every classifier is an instance of the same rule, in that case the ensemble would stay at 51% accuracy). This is the reason we usually try to ensemble a lot of different algorithms, hoping for maximally „independent“ classifiers. $\endgroup$
    – AlexR
    Jul 7, 2020 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.