What is the difference between horizontal and vertical ensemble?

Question

I am looking at different ways to do model ensembling and I came across the terms horizontal and vertical blending/ensembling but it is not well defined.

My questions will be:

1. What is the difference between horizontal and vertical blending?
2. In which cases which one is better?
3. Are there any python implementations for them?

Answer 1

I think the most highly-referenced source for these terms is Horizontal and Vertical Ensemble with Deep Representation for Classification (Xe, Xu, Chuang 2013). That would be the best place to get a technical answer to your first question. For purposes of searching, you could also look into "Stacking" as a synonym for Vertical ensembling. I will provide some intuition here:

1. Ensembling is, to over-simplify, adding new content to your data based on results from other or intermediate modeling results. In this case, "horizontal" and "vertical" are references to where in your dataset you add this content. Horizontal ensembling takes the results from one model (such as the predicted classification) and appends it as a new column to your data. In the case of a binary classifier result, this would effectively be adding a column that says "A different model classified this row as a 1 or 0 (A or B, red or green, etc)." The final model would use this information in its own process as a new feature. Vertical ensembling adds new predicted values (as rows, if you want to think about a dataframe paradigm) that are generated by a model. To be clear, Horizontal ensembling adds what some external/interediate model thinks is the right classification while vertical ensembling adds what the model predicts as a value.

Consider a problem that is trying to predict whether the image is "cat" or "dog". Based on the features in the row, a model can say "cat", which gets added to that row when the Horizontal ensemble adds the results of a different classifier. Think of that as an extra vote that the image is of a cat. Or a Vertical ensembling can add new information from that external model for the predicted probabilities of the image being "cat" or "dog", maybe as a tuple: (0.73, 0.27).

2. Give the article a good read to see when you might prefer one over the other, and the statistical issues that arise in both cases. And, as always, keep the No Free Lunch Theorem in mind.

3. Ensembling can refer to multiple things. Boosting methods can be considered ensembling, since these algorithms are doing intermediate calculations to see what learners are performing better in order to shift weights in the modeling. These methods are built into the algorithms themselves, so I would look into scikit learn and its ensembling methods to get a sense of what's going on. To your specific concern, however, you can look through Github repos for ensemble learning.