Model blending -- by which I mean creating multiple sets of predictions from models that have the same dependent variable and the same or similar independent variable candidates, as opposed to model stacking -- is a popular way of creating ensembles of Machine Learning models.

For example:

Y = regression_predictions * .5 + tree_predictions * .5

While this approach is useful in a wide variety of use cases, a good example is Kaggle competitions. In these competitions you have:

  • Labeled training data
  • Unlabeled prediction (test) data
  • A certain number of allowed submissions per day, often 5, wherein you submit your predictions and receive instant scoring based on a specified evaluation metric like RMSE

My question is if there is an statistical or logical methodological framework that can guide the testing of various blends of your models?

That is, can some algorithm or methodology be applied to the process of deciding how to alter the blend of predictions?

Ideally, such a framework would also guide the decision of when to add or remove a set predictions, though even a framework which assumes a given number of models would be highly useful.


1 Answer 1


This is normally called "Ensemble learning": wiki link

Bayesian Model Combination is a popular modern technique which usually delivers better results than Bayesian Model averaging, on which it was based. The latter had a tendency to give almost all the weight to one model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.