Model blending -- by which I mean creating multiple sets of predictions from models that have the same dependent variable and the same or similar independent variable candidates, as opposed to model stacking -- is a popular way of creating ensembles of Machine Learning models.
Y = regression_predictions * .5 + tree_predictions * .5
While this approach is useful in a wide variety of use cases, a good example is Kaggle competitions. In these competitions you have:
- Labeled training data
- Unlabeled prediction (test) data
- A certain number of allowed submissions per day, often 5, wherein you submit your predictions and receive instant scoring based on a specified evaluation metric like RMSE
My question is if there is an statistical or logical methodological framework that can guide the testing of various blends of your models?
That is, can some algorithm or methodology be applied to the process of deciding how to alter the blend of predictions?
Ideally, such a framework would also guide the decision of when to add or remove a set predictions, though even a framework which assumes a given number of models would be highly useful.