Questions on ensemble technique in machine learning

Question

I am studying the ensemble machine learning and when I read some articles online, I encountered 2 questions.

1.

In this article, it mentions

Instead, model 2 may have a better overall performance on all the data points, but it has worse performance on the very set of points where model 1 is better. The idea is to combine these two models where they perform the best. This is why creating out-of-sample predictions have a higher chance of capturing distinct regions where each model performs the best.

But I still cannot get the point, why not train all training data can avoid the problem?

2.

From this article, in the prediction section, it mentions

Simply, for a given input data point, all we need to do is to pass it through the M base-learners and get M number of predictions, and send those M predictions through the meta-learner as inputs

But in the training process, we use k -fold train data to train M base-learner, so should I also train M base-learner based on all train data for the input to predict?

Answer 1

Instead, model 2 may have a better overall performance on all the data points, but it has worse performance on the very set of points where model 1 is better. The idea is to combine these two models where they perform the best. This is why creating out-of-sample predictions have a higher chance of capturing distinct regions where each model performs the best.

It's not about training on all the data or not. Both models trained on all the data. But each of them is better than the other at different points. If I and my older brother are tying to guess the exact year of a song, I will do better in 90s songs and he in 80s songs - it's not a perfect analogy but you get the point - imagine my brain just can't process 80s songs, and his can't process 90s songs. The best is to deploy us both knowing we each have learnt different regions of the input space better.

Simply, for a given input data point, all we need to do is to pass it through the M base-learners and get M number of predictions, and send those M predictions through the meta-learner as inputs

k-fold is still just one learner. But you're training multiple times to chose parameters that minimize error in the left-out fold. This is like training only me on all the songs showing me k-1 folds of data, and I calibrate my internal model the best I can... but I'll still never be very good at those 80s songs. I'm just one base learner whose functional form (my brain) isn't fit for those songs. If we could bring the second learner along, that would improve things.

Answer 2

1- The idea of ensemble methods is to reduce variance mostly, which means "overfitting". The idea behind it is that we train different models ( in this case 2 models ) that don't necesseraly see all the training data points, so it won't overfit on it ( method commonly called as bagging ). Now for the prediction we can do a vote for classification problems or mean value for regression problems or even a whole learner for that. That ensures that the model will be stable through training and test phase.

So if we train on all training data we may get overfitting and thus low accuracy on test cases.

2- have a look at this one :Why use both validation set and test set?