let me begin by saying that I understand how to build a stacked ensemble by using cross-validation to generate out-of-fold predictions for the base learners to generate meta-features. My question is about the methodology when cross-validating the entire stacked ensemble to check generalization error.
To eliminate any confusion, I'm going to call the cross-validation to generate out of fold predictions for the base learner CV A, while I'll call the cross-validation of the entire stacking ensemble CV B.
When I do CV B, is it valid to do CV A just once and use those out of fold predictions for the entire CV B process? Or do I have to keep doing CV A and generate new out of fold predictions during each fold of CV B?
Normally, I'd think that there'd be some data leakage in the first method, but one could also reason out that since the out of fold predictions are taken, well, out of fold, that issue is taken care of. The main reason I'm asking this is because doing the second method would surely remove any data leakage but there would be an order of magnitude of additional computational complexity involved.