How does the stacking works?

Question

Suppose that I have $n$ trained weak base models: $m_1, m_2, ..., m_n$ As I understand after training that models we get their predictions on validation dataset, let's consider single element of sample. Predictions of models $y_1, y_2, ..., y_n$ (meta-features). Then we add true label to our data $y_1, ..., y_n, y_{true}$. And on that meta-features we train our meta-model. Here I have several questions.

1. If it's supervised learning task what would be label to train meta model? $y_{true}$? If the answer is yes, I have no more questions. Otherwise I have no clue how we will make final predictions on holdout dataset.

Thanks for your help!

Answer 1

You are partially correct:

• Yes, the meta-model is trained to predict the label $y_{true}$.
• But $y_{true}$ is not used as input feature for the meta model. This would not work in any application / real-world scenario, where $y_{true}$ is not known and the (meta-)model is actually used to predict it.
• Optionally, the meta model can also get the base features (i.e. the input to the models $m_1, \ldots, m_n$) as additional input. This can be helpful, if the base features help to decide which model is more reliable in a particular case.