I have seen that consensus of classifiers (taking say 5 separate classifiers) and obtaining the final labeling of the unknown sample based on the voting method (whichever class gets the predicted the most is the class of the unknown sample) works better than taking a single classifier for predicting the class of a sample. Why so? Any articles which show why this happens?
For a very simple example, imagine you have three independent classifiers that each have 60% accuracy. If you use any one of them to classify a random sample, you have a 60% chance of getting it right.
Now use an ensemble classifier that takes the majority vote. Below are the probabilities that exactly N of the individual classifiers are correct.
0 correct - 6.4%
1 correct - 28.8%
2 correct - 43.2%
3 correct - 21.6%
Your ensemble classifier has the correct output when 2 or 3 of the component classifiers are correct, which happens 64.8% of the time. Your ensemble classifier outperforms each of the individual component classifiers! You'll get this advantage by combining any classifiers that perform better than random.
A very straightforward explanation is they can reduce both bias and variance to limit overfitting. Below points from that link:
- Reduce bias, each model learns something slightly different and by taking an ensemble, can cancel out any crazy/spurious things one model learns.
- Reduce variance, each model is suscptible to slightly different noise, and so by taking an ensemble you can diversify away the noise.
- Reduce overfitting, it is harder for an ensemble to overfit, since they don't necessarily all see the same data and the individual models learn different things
Search around for "ensemble techniques", which covers this sort of thing. Random Forests are probably the best known kind.