In stacked generalization, if I understood well, we divide the training set into train/test set. We use train set to train M models, and make predictions on test set. Then we use the predictions as input of a new model. Thus, the new training set will have M features corresponding to the M models predictions. Finally, we use the last model to make the final predictions. First, is my understanding correct ? If so, how is it possible to use the last model to make predictions as it has different features.


1 Answer 1


You have created a model pipeline and must run all trained models ("lower level" ones first) in order to make a prediction on new data using the stack.

With test data set, it is slightly easier, since you can store the predictions from the "level 1" models when testing them, and only run the final model across this stored data.

In addition to your brief description, usually to avoid bias from re-using training data, you would use k-fold cross-validation or similar mechanism, and your training data for the final model should be the cv predictions from each model. You do not want to use the training predictions from those models, because they are likely to be overfit whilst "level 1" test and production predictions will not be, and this would introduce population differences between train and test data in your "level 2" model.

It is also quite a common variation to use the M new features from your "level 1" models alongside some or all of the original features. This gives the meta model more data to base its decision on when deciding the relative weights between the first stage models (assuming this top-level model is non-linear).

  • $\begingroup$ Thanks Neil Slater. In the different examples that I've seen, the training set used to train the level 2 model has M columns and N rows (M: number of models, N: number of training samples). Is that correct ? If so, how can the production set be used as it is of different shape. $\endgroup$
    – Spider
    Commented Jun 14, 2017 at 12:35
  • $\begingroup$ @Spider: The number of samples processed in a single prediction is usually allowed to change freely between training and prediction (and sometimes even during training). I don't know of any supervised learning frameworks where this is not the case. Otherwise you could not predict individual examples when needed. If you are confused about that, then perhaps ask another question, as it is not related to stacking. $\endgroup$ Commented Jun 14, 2017 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.