I have few questions for which I could not extract answers from text books and online tutorials. Therefore, will be extremely grateful if the following points are clarified.

1) If I want to apply SVM, MLP and decision trees and combine the prediction results from these learners. So, I will have a mixture of ensemble of SVM, MLP and Decision trees OR I can use one learner only and have an ensemble of decision trees, an ensemble of SVMS and an ensemble of MLPs. Is my understanding correct? How do I combine the prediction results from different models?

2) In ensemble learning be it homogeneous and heterogeneous, each of the learner is trained on the same data set? Is this technique known as Bagging or simply ensemble learning? Would I be using the same subset of data for training each learner or different?

2) I have read in many tutorials that bagging and boosting is performed on similar learners. But ensemble learning can be homogeneous and heterogeneous. As a result, can bagging and boosting be performed on different learner types?

3) In Boosting, the learners are trained sequentially and therefore often only a learner of one kind is used as it is probably difficult to know if the sequence influences the learning procedure. Am I correct?

4) How is stacking different from the rest?


Here's my two cents:

  1. Yes, if you want to perform enseble each model should be trained on the same set of data. The technique you described is stacking, because you "stack" each prediction and simply use the majority. Ensemble Learning, to me, si more "complex", infact you can combine the different predictions from k models via linear combination, where you estimate the coefficients of the combination itself.

More precisely, you have k prediction for each of your test observations, from k models (ie: 3 or more), and you need to find the best combination (for each obs.) of these, to have the best "accuracy" (any kind of metric).

Keep in mind that it's best to combine models that have low or even opposite correlation, so that you "diversify" your predictions, that would mean better results.

Often peole train lots and lots of different model, than use a simple Neural Network using the predictions as inputs, and the true labels from the test set as the truth, just as a normal class model would do.

See the papers in the References here as a starting point.

  1. Bagging is basically a special case of stacking via majority vote, see:


You propose to the same type of model (ie: decision trees) the train dataset multiple times, making new different extraction, with replacement. This is like submitting new entries, but basically you are using the same data all over. Then you average all the predictions again, this helps with overfitting but it's not always the case, because you are using the same model, and so each si highly correlated with the other.

  1. Yes, in boosting, just like bagging you use the same model (usually a weak learner). At each stage, the learner learns from the errors made at the previus stage.

Here an excellent visual example of what's going on.

To be clear, stacking and ensemble learning can be used with lots of different models on the same train set (it's even better if the model in the "ensemble" are quite different). You can combine each prediciton with different techniques, from simple average, to the more complex use of Neural Networks.

Bagging and Boosting are based on the use of the same model, one using the same obs. multiple times, the latter uses the same weak learner on the same data, but each makes predictions using the errors made by the previous one, in a loop.

Hope this helps you a little.

  • $\begingroup$ Explained very lucidly, got it. Thank you. WOuld it be possible for you to answer another question asked here datascience.stackexchange.com/questions/33582/… It would be of immense help. To summarize, I am asking that for an imbalanced dataset situation suppose I create 5 different sets of balanced data and on each set I am doing training by using k fold cross-validation. Inside the cross validation loop, I am also testing using the other dataset not used in the training. $\endgroup$ – Ria George Jun 25 '18 at 18:20
  • $\begingroup$ Then, outside the cross-validation loop should I test the model on an unseen new data that is not balanced and has not been used in training? $\endgroup$ – Ria George Jun 25 '18 at 18:23
  • $\begingroup$ The final test must always be on new data, and with the true ratio between the labels, otherwise it won't be reliable. $\endgroup$ – RLave Jun 26 '18 at 6:27
  • $\begingroup$ If the performance is not satisfactory, then do we select a different learner or start the training procedure again with a different setting of say a kernel in the case of SVM? thank you for your suggestions. $\endgroup$ – Ria George Jun 27 '18 at 3:38
  • 1
    $\begingroup$ This part is always very hard, and each dataset is different. $\endgroup$ – RLave Jun 27 '18 at 6:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.