4
$\begingroup$

let me begin by saying that I understand how to build a stacked ensemble by using cross-validation to generate out-of-fold predictions for the base learners to generate meta-features. My question is about the methodology when cross-validating the entire stacked ensemble to check generalization error.

To eliminate any confusion, I'm going to call the cross-validation to generate out of fold predictions for the base learner CV A, while I'll call the cross-validation of the entire stacking ensemble CV B.

When I do CV B, is it valid to do CV A just once and use those out of fold predictions for the entire CV B process? Or do I have to keep doing CV A and generate new out of fold predictions during each fold of CV B?

Normally, I'd think that there'd be some data leakage in the first method, but one could also reason out that since the out of fold predictions are taken, well, out of fold, that issue is taken care of. The main reason I'm asking this is because doing the second method would surely remove any data leakage but there would be an order of magnitude of additional computational complexity involved.

$\endgroup$
2
$\begingroup$

I posted this same question on Reddit and someone was kind enough to answer

By https://www.reddit.com/user/patrickSwayzeNU

This

When I do CV B, is it valid to do CV A just once and use those out of fold predictions for the entire CV B process?

Normally, I'd think that there'd be some data leakage in the first method, but one could also reason out that since the out of fold predictions are taken, well, out of fold, that issue is taken care of.

Yes, your data set created by CV A is now "good as new".

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.