Ensembling vs clustering in machine learning

Question

Following the raise of ensembling (e.g ensembling of xgboost learners) after its recurrent wins in Kaggle competitions, using it has become a tradition in machine learning. However, some argue that clustering is a form of ensembling where we apply clustering on a data set first, then applying a learning algorithm (to do classification or regression) on each of the resulting clusters.

I would like to understand how clustering is different from ensembling?

EDIT

I understand that clustering can be similar to the idea of ensembling if we simply consider "homogeneous ensembling" where we train the same model on each of the clusters. But I am looking for potential limitations of clustering where using heterogeneous ensembles (different learns for all the data) is the "only" way to go about the problem.

Answer 1

Short answer: Ensembling and clustering are completely unrelated techniques.

Ensembling: Combine the strengths of many diverse models. Ensembles generally do not involve training models on separate sets of data -- it's the models themselves that are different. Generally the more diverse the models the better. For example, an ensemble might comprise the following models: a support vector machine (SVM), a random forest, a gradient boosting model (GBM), logistic regression, k-nearest-neighbors, and a neural network. On Kaggle, I have also seen top competitors create ensembles comprising the same type of model (e.g., random forests) but with a variety of tuning parameters (e.g., one with 100 trees, another with 250 trees, etc). Clustering could be one of the models included in an ensemble. Ensembles are very effective (although extremely complex to operationalize) and can generally improve your error about 2-5%.

Clustering: Divide the data into mutually exclusive groups with the hope that your cluster-specific models can reduce error by specializing, except in the case of fuzzy clustering where the groups need not be mutually exclusive. At financial companies that build credit risk models, these cluster-specialized models are sometimes called "sub-models" in the sense that they predict the same target variable and therefore fit under an abstract/figurative parent model. This makes model risk reporting easier in the sense that it's simpler to refer to model 1 then model 1a, 1b, 1c, 1d. Clustering in this context is generally based on business knowledge and not a formal clustering technique such as k-means or other types of centroid-based clustering. For supervised learning tasks, formal clustering is generally considered a pretty poor technique because measuring the quality of the clusters is subjective. Your clusters can also vary dramatically depending on how you initialize the centers and how many clusters you specify. In practice, clustering is used much less frequently than ensembling.

Answer 2

As far as I understand your question, the difference between those two approaches lies in the field of limits you apply to the model. During ensemble learning, say xgboost, you will be training multiple boosted trees models, each of which would be trained on a (random) subset of features and (random) subset of species from your dataset. That way, you will get N classifiers, for example, each of which would have slightly different experience about your problem, but every one of them would be trained on randomly limited (with no specific limit) chunk of data.

If you apply clustering (for any reason, be it diversity of classifiers or reduction of dataset to fit it into memory) before training, you fist rely on a clustering algorithm to find some sort of communities in your dataset and then you train separate classifiers for every cluster. How they are used in your example further, I am not completely sure about, but a concise way would be assignment of a particular cluster for a new data and using 'right' classifier for that instance.

For a real-world example, take loan approval problem. In ensemble learning you will have three persons sitting in the same office, talking to a constant flow of customers in need of money, and every credit approver would have similar experience, but will tend to look into particular data they are familiar with, like job, marital status or any other 'feature'.

In clustering, you will have three persons approving loans, but one will be working strictly with farmers, other will be looking into cases of recently fired people and third would be dealing with students in urge to buy a new smartphone. All three will be working on the same problem, but their experience will be focused on a subset of people and a student-handler wouldn't be a good choice to make a decision on a farmer loan since he has no experience with appraising their financial situation.

Clustering approach makes more sense when you have a diverse data which would contradict itself in terms of training (load approval criterion would be different for gender-based, marital status based and so on, and married persons would be approved with higher probability than single once due to responsibilities and stuff even if their income would be lower than single person's) - but for every cluster you would have a single working model, other-cluster model would provide random results since it hasn't been trained on other-cluster data.

Ensemble learning works great in a way that you get a set of models with different 'experience' about your problem. Each classifier would have its own precision/recall/general performance metric, and if three classifiers with different feature and example set would agree on an outcome, generally it is safe to say that if three classifiers say you it is an apple, it should be an apple (because, error rate of 1% for each classifier would give you 1%*1%*1% = 0.0001% chance that all three classifiers are incorrect).

The arithmetic of joint classifier error rate isn't that straightforward since usually ensemble sub-models (talking about xgboost here) aren't independent as they share some features, but general idea is like that.

Answer 3

It depends on how you combine results.

Many ensemble techniques will either:

1. train the same classifier, but on different parts of the data
2. train different classifiers on the full data
3. vary both: classifier and data subsets

In either situation, you afterwards have to combine the results; usually with some form of majority voting. So if 2 classifiers return "A", 1 classifier returns "B", then the outcome is "A".

To get a good outcome, every member needs to be better than random; to improve over the individual results they must not be too similar.

You could use clustering for the first approach (to get different parts of the data). But the problem is that these parts are not independent, and too biased. You would usually want each classifier to know "a little bit of everything". By withholding some part of the data you prevent them from overfitting the same way. For this, random is usually best. If you do clustering, there is a chance you get 1 classifier that thinks everything is "A", 1 that thinks everything is "B", and 1 that thinks everything is "C". You even encourage them to overfit! So you always get the result 1 A, 1 B, 1 C = no majority.