How to do clustering assuring more than one class per cluster?

Question

I have a dataset with 4 classes and i'm trying to use an ensemble model where each base classifier trains with a portion of data. To distribute data along the classifiers, i am using KMeans algorithm. The problem is that, in some cases, a cluster has just one class and the classifier fitted to this data performs poorly in prediction. Besides that, having just one class per cluster prevents me from using some structures (e.g. Logistic Regression, SVMs, etc).

I would like to know if there's any clustering technique which can keep a fair distribution of data, considering more than one class per cluster. I know that clustering is unsupervised learning and doesn't take into account the class of each sample, but i don't have any other idea rather than using another technique instead of KMeans.

Thanks

Answer 1

This is a very strange design:

• The goal is to train an ensemble classification model. In general there is no strong reason to use only subsets of the data to train the individual learners, let alone to use a strict partition of the data. It might make sense to train different learners with different subsets in order to make the final model more stable, but at least the sampling should be made with replacement, i.e. allowing an instance to be used for several learners. Forcing every instance to be used in only one learner is very likely to make most or all the individual learners very weak, in turn making the final model certainly not as a good as a simple classifier using all the data.
• Using clustering to prepare the subsets is even stranger, I cannot think of any rationale for this: not only it can be expected that some clusters would correspond to specific classes, it's actually a good thing: it means that the data contains some patterns which correspond more or less to the classes. If there were no such pattern the classification problem would be unsolvable. It's counter-productive to use the clusters as training subsets for the individual learners, since one wants these models to be able to distinguish the classes. So they need examples of several classes, and preferably not a completely biased sample otherwise they can't do their job properly. If you really want to use subsets of data, the subsets should simply be picked randomly, not obtained by clustering.
• In case you really want to use clustering, the resulting cluster for every instance can be used as a feature for the classification stage. Not as a way to obtain subsets of data.

There is no clustering method which lets you specify constraints based on class, because that's not at all the logic of a clustering algorithm. What you could use is stratified sampling, i.e. picking instances randomly class by class in order to make sure that every subset contains the same distribution across classes.