Ensembling is getting more and more popular. I understand that there are in general three big fields of ensembling, bagging, boosting and stacking.

My question is that does the ensembling always at least increase the performance in practice? I guess mathematically, it is not true. I am jsut asking a real life situation.

For example, I could train 10 base learner, and then stack them with another learner, which is at 2nd level. Does this 2nd-level learner always outperform the best of the base learners in practice?

  • $\begingroup$ stats.stackexchange.com/questions/109831/… $\endgroup$ – HelloWorld Feb 23 '17 at 22:21
  • $\begingroup$ Are you still looking for an answer? $\endgroup$ – Toros91 Apr 19 '18 at 5:37
  • $\begingroup$ @Toros91 Yup... $\endgroup$ – WCMC Apr 19 '18 at 15:03
  • $\begingroup$ Ohk would that tomorrow in the first hour. $\endgroup$ – Toros91 Apr 19 '18 at 15:34

The short answer is no.

I have worked on several projects that evaluated an ensemble of several classifiers versus the classifies themselves. In some cases the precision and recall was better with the ensemble, but more often, it was not. That's not to say it's not worth investigating. But sometimes, there is one model that does a reasonable job of classifying data, but it can get drowned out in an ensemble. Perhaps a weighted ensemble might improve the results, but its not a clear cut approach to improving the performance.

In practice, I would try several models and then try an ensemble of the models. If the ensemble is the best, however you define best, go with it. But sometimes it is easier to just pick the best base model and then figure out how to tune that model.

  • $\begingroup$ Hi, when you measure the performance, are you talking about the performance of cross-validation or performance on a validation set that is completely blind when training the model? I am guessing that the ensemble should (always?) be better in terms of preventing overfitting. $\endgroup$ – WCMC May 21 '18 at 4:18
  • $\begingroup$ Well, I would suggest that you consider the performance of both. CV is beneficial because it tells you what performance you may expect to achieve in the long run. When you actually produce a final model, you should produce it on all available data. $\endgroup$ – Skiddles May 21 '18 at 14:19
  • $\begingroup$ Sorry. Timed out. That said, it seems intuitive that the ensemble should be better, and maybe with the right weighting it can be, or if the weak classifies do a reasonable job it can, but sometimes, there are just a bunch of poorly performing models that outvote a decent to good model. It's totally worth trying ensembles, but they are not a guarantee of better performance. $\endgroup$ – Skiddles May 21 '18 at 14:45

As you said, you cannot prove mathematically that esembling increases performance, but it generally does. That's reason why gradient boosting and random forests are so popular in kaggle competitions, because they outperform what a decision tree can learn in many ways.

As a curiosity, even Neural Networks can be used as "weak" learners, as can be seen in https://arxiv.org/abs/1704.00109. So, ensembling is a very powerful technique that can be applied in many areas of machine learning. The main problem is that ensembles are not easily interpretable, being way more black-boxy than its weak learners.


Under Ensemble you can use Majority Votes, Average, Weights etc to get the final outcome from Ensemble model. To understand it better you can go through this Link, explained well by Alexander.

Now, let us consider that you have 3 models which has an accuracy of 65-70%. Now by stacking these 3 models there is very high chance that you models accuracy would increase. In another scenario you have 3 models model-1: 95%, model-2: 55%, model-3: 45% accuracy, then if you stack them then there is a very good chance it can worsen the result.

Conclusion, it all depends on the individual models performance, Ensemble performs well when you combine moderately performing models.

Technically there is no proof saying that this method is suitable for this scenario but trail and error might help you to get good results. It is subjective to the business scenario. Similarly, for bagging and boosting.

In my experience with Bagging when the model accuracy is bad, I tried using bagging to fit the data better but EOD training accuracy(20% to 10% approx) was decreased but test accuracy was worsened(11% to 20% approx). So, you have to decide which suits your business problem better and take it forward.

  • $\begingroup$ was my answer helpful? $\endgroup$ – Toros91 May 4 '18 at 7:40

If your individual classifiers are better than random guessing , i.e their error rate is less than 0.5 , then ensemble of these classifiers will lead to an increase in performance , i.e , a drop in error rate.

You can refer to Combining Different Models for Ensemble Learning chapter in Sebastian Raschka - Python Machine Learning Book for a mathematical understanding of the same.

  • $\begingroup$ The errors must also be uncorrelated! $\endgroup$ – jonnor Jun 29 '18 at 10:57
  • $\begingroup$ @jonnor , True . Btw , is there a way to check whether the errors are uncorrelated or not? $\endgroup$ – Faraz Gerrard Jamal Jun 29 '18 at 11:33
  • $\begingroup$ Run the individual classifier on test set data and see to how much they agree with each other $\endgroup$ – jonnor Jul 2 '18 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.