Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
I am trying to implement stacking model for a ML problem and having hard time figuring out the cross validation strategy. So far I have used 10-fold cross validation for all my models and would like continue using that stacking as well. Here's what I came up with but not sure if it makes sense,
At each iteration of 10-fold CV, you will have 9 folds for training (training dataset) and 1 fold for testing (testing dataset).
Divide the training dataset into 3 parts - F1, F2 and F3.
Train base classifiers on F1, use F2 for early stopping and get out of fold predictions on F3 -> F3' (F3' is the set of predictions made by base-classifiers on F3)
Train base classifiers on F2, use F3 for early stopping and get out of fold predictions on F1 -> F1'
Train base classifiers on F3, use F1 for early stopping and get out of fold predictions on F2 -> F2'
Train meta-classifier on (F1' + F2' + F3'), train base-classifiers on any two folds and use remaining fold for early-stopping.