I was reading this article talking about ensemble models. I was interested in the max voting model using 3 base learners. However, I am a little confused about the process. Currently, I'm thinking it goes like this: I have a training and testing sets. All three models are trained on the training set individually and finally at the end I combine the 3 models and do max voting on the testing set and see the results. Instead, should the original training set be divided such that each base learner does not see the same training data?

  • 2
    $\begingroup$ The splitting to training & test data takes place very early in the modeling process, before you have even decided what model(s) you will employ, so it is not at all affected by the model choice (ensemble or not). $\endgroup$
    – desertnaut
    Commented Aug 2, 2022 at 20:40

2 Answers 2


When ensembling, you need some method of introducing diversity into your models (otherwise all your models will make the same prediction, so ensembling them won't improve the results). Using different training data for each model is one way of introducing this diversity. A common method is to use bootstrapping or bagging, where you randomly sample (with replacement) from your training data. This is what the random forest algorithm does (although it also randomly selects the features for even more diversity). As pointed out by @desertnaut, you do your initial test/training split first, then form your ensemble training sets using only the training data.

However, there are several other ways to introduce diversity into your models:

  1. Boosting - where the models are trained in sequence. Each model re-weights the training samples, increasing the weight of samples the previous model classified incorrectly and decreasing the weight of those previously classified correctly. This is how AdaBoost works.
  2. Use different classifiers - e.g. if you want 3 learners you could ensemble a logistic regression model, an SVM and a neural network.
  3. Use different architectures or hyper-parameters, so use SVMs with different kernels or different sized neural networks.
  4. If using neural networks, initialise each network differently, so that when trained, each model converges to a different solution.
  • $\begingroup$ Currently, I am trying to use 3 different classifiers. Would that mean this act is enough to introduce diversity instead of also having to split the initial training set? $\endgroup$
    – ddd
    Commented Aug 3, 2022 at 2:47
  • $\begingroup$ @ddd - it should be, as long as you get different results from each classifier - i.e. each classifier mis-classifies different exemplars. $\endgroup$
    – Lynn
    Commented Aug 3, 2022 at 5:51

You should make hypothesis about data.

Hypothesis 1: Data has subgroups

In first case , You need to split and feed part in three models. Final model outcome will be based on the group instance belongs. The output will correspond to one among three models

Hypothesis 2: data does not has sub groups Then feed all data in all models . Final Model outcome will be aggregate of outcome of base models


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.