I am trying to implement stacking model for a ML problem and having hard time figuring out the cross validation strategy. So far I have used 10-fold cross validation for all my models and would like continue using that stacking as well. Here's what I came up with but not sure if it makes sense,

  1. At each iteration of 10-fold CV, you will have 9 folds for training (training dataset) and 1 fold for testing (testing dataset).

  2. Divide the training dataset into 3 parts - F1, F2 and F3.

  3. Train base classifiers on F1, use F2 for early stopping and get out of fold predictions on F3 -> F3' (F3' is the set of predictions made by base-classifiers on F3)

  4. Train base classifiers on F2, use F3 for early stopping and get out of fold predictions on F1 -> F1'

  5. Train base classifiers on F3, use F1 for early stopping and get out of fold predictions on F2 -> F2'

  6. Train meta-classifier on (F1' + F2' + F3'), train base-classifiers on any two folds and use remaining fold for early-stopping.

  7. Validate meta-classifier on test dataset

Repeat these steps for every fold in 10 fold CV.


1 Answer 1


In the diagram below that you can find here, it is shown the general approach to train a stacking model:

Thes steps are:

Given $T$ a train set of shape $mxp$

  1. Divide $T$ into $k$ folds
  2. Fit classifiers $M_1$, $M_2$,...,$M_n$ on $k-1$ and predict on fold $k$
  3. Save the predictions of the $M$ classifiers on fold $k$ (predictions on step 2)
  4. Repeat steps 2 and 3 $k$ times

After those steps, you will end up with a dataset $D$ of shape $mxn$

  1. Train a model $\Phi$ (meta model) on $D$ and retrain the $M$ different classifiers on $T$

Then you can use $\Phi$ to make predictions on unseen data

enter image description here

Hope it helps!

  • $\begingroup$ Thanks for the answer. My base-classifiers are neural nets so I need a validation set for early stopping. Can I use same set for validation of base-classifier and building out-of-fold predictions for meta-classifiers? $\endgroup$
    – LoneWolf
    Feb 11, 2022 at 3:46
  • $\begingroup$ As far as I know, this approach is model agnostic, so you should be ok with neural networks as first-level classifiers. You could also have a look of this post $\endgroup$
    – Multivac
    Feb 11, 2022 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.