I am using the stacked generalization scheme to combine the predictions from different machine learning models (input models from now on).

I am currently calculating the prediction interval for each of these input models by means of bootstrapping. However, I cannot do the same with the stacked model, since it won't bootstrap the original data but rather the input models' outcome, and therefore the prediction interval would very likely be underestimated.

I believe an alternative could be to retain each paired bootstrapped prediction from the input models, and then build a stacked model on each of these, so that in the end I would end up with a distribution of stacked predictions which would (hopefully?) account for the input models' variability. This seems too tedious and I am not even sure it would be the right thing to do.

I was wondering if there's an alternative/better way to achieve this. I am thinking about other means to calculate the uncertainty (Conformal Prediction comes to mind), and if I can somehow exploit some statistical characteristics which I am now not considering with my current method (e.g. given some assumptions and a confidence interval maybe I can work out something from the prediction bands?). I am not a statistician and I am kind of stuck at the moment. The only resource on the topic I could find so far was this, but it does not provide a solution unfortunately.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.