I have a dataset with 4 classes and i'm trying to use an ensemble model where each base classifier trains with a portion of data. To distribute data along the classifiers, i am using KMeans algorithm. The problem is that, in some cases, a cluster has just one class and the classifier fitted to this data performs poorly in prediction. Besides that, having just one class per cluster prevents me from using some structures (e.g. Logistic Regression, SVMs, etc).
I would like to know if there's any clustering technique which can keep a fair distribution of data, considering more than one class per cluster. I know that clustering is unsupervised learning and doesn't take into account the class of each sample, but i don't have any other idea rather than using another technique instead of KMeans.