Stacking neural nets with cross validation

Question

I am trying to implement stacking model for a ML problem and having hard time figuring out the cross validation strategy. So far I have used 10-fold cross validation for all my models and would like continue using that stacking as well. Here's what I came up with but not sure if it makes sense,

1. At each iteration of 10-fold CV, you will have 9 folds for training (training dataset) and 1 fold for testing (testing dataset).

2. Divide the training dataset into 3 parts - F1, F2 and F3.

3. Train base classifiers on F1, use F2 for early stopping and get out of fold predictions on F3 -> F3' (F3' is the set of predictions made by base-classifiers on F3)

4. Train base classifiers on F2, use F3 for early stopping and get out of fold predictions on F1 -> F1'

5. Train base classifiers on F3, use F1 for early stopping and get out of fold predictions on F2 -> F2'

6. Train meta-classifier on (F1' + F2' + F3'), train base-classifiers on any two folds and use remaining fold for early-stopping.

7. Validate meta-classifier on test dataset

Repeat these steps for every fold in 10 fold CV.

Answer 1

In the diagram below that you can find here, it is shown the general approach to train a stacking model:

Thes steps are:

Given $T$ a train set of shape $mxp$

1. Divide $T$ into $k$ folds
2. Fit classifiers $M_1$, $M_2$,...,$M_n$ on $k-1$ and predict on fold $k$
3. Save the predictions of the $M$ classifiers on fold $k$ (predictions on step 2)
4. Repeat steps 2 and 3 $k$ times

After those steps, you will end up with a dataset $D$ of shape $mxn$

5. Train a model $\Phi$ (meta model) on $D$ and retrain the $M$ different classifiers on $T$

Then you can use $\Phi$ to make predictions on unseen data

Hope it helps!