I was reading this article talking about ensemble models. I was interested in the max voting model using 3 base learners. However, I am a little confused about the process. Currently, I'm thinking it goes like this: I have a training and testing sets. All three models are trained on the training set individually and finally at the end I combine the 3 models and do max voting on the testing set and see the results. Instead, should the original training set be divided such that each base learner does not see the same training data?
When ensembling, you need some method of introducing diversity into your models (otherwise all your models will make the same prediction, so ensembling them won't improve the results). Using different training data for each model is one way of introducing this diversity. A common method is to use bootstrapping or bagging, where you randomly sample (with replacement) from your training data. This is what the random forest algorithm does (although it also randomly selects the features for even more diversity). As pointed out by @desertnaut, you do your initial test/training split first, then form your ensemble training sets using only the training data.
However, there are several other ways to introduce diversity into your models:
- Boosting - where the models are trained in sequence. Each model re-weights the training samples, increasing the weight of samples the previous model classified incorrectly and decreasing the weight of those previously classified correctly. This is how AdaBoost works.
- Use different classifiers - e.g. if you want 3 learners you could ensemble a logistic regression model, an SVM and a neural network.
- Use different architectures or hyper-parameters, so use SVMs with different kernels or different sized neural networks.
- If using neural networks, initialise each network differently, so that when trained, each model converges to a different solution.
You should make hypothesis about data.
Hypothesis 1: Data has subgroups
In first case , You need to split and feed part in three models. Final model outcome will be based on the group instance belongs. The output will correspond to one among three models
Hypothesis 2: data does not has sub groups Then feed all data in all models . Final Model outcome will be aggregate of outcome of base models