5
$\begingroup$

I'm learning the GMM clustering algorithm. I don't understand how it can used as a classifier. Here are my thought:

1) GMM is an unsupervised ML algorithm. At least that's how sklearn categorizes it.

2) Unsupervised methods can cluster data, but can't make predictions.

However, sklearn's user guide clearly applid GMM as a classifier to the iris dataset.

If I have to guess, maybe after clustering, each cluster is assigned to a class label based on some kind of majority voting. However, I can't find any documentation. Could someone shed more light on this process from unsupervised to supervised learning?


A related question: when using GMM as a classifier, is it common practice to simply make n_components=n_classes, instead of checking AIC, BIC, etc.?

$\endgroup$
  • 1
    $\begingroup$ If you know n_classes, then you are using supervised information. And you assume the labels are optimal. In real data, you likely don't know the number of classes, or you may be looking for alternate (sub-) structures. $\endgroup$ – Has QUIT--Anony-Mousse Feb 21 '19 at 15:27
4
$\begingroup$

Some unsupervised models can make predictions, but not ones that necessarily match the original class labels. Once a GaussianMixture model has been fitted, it can predict which of the clusters a new example belongs to. This is exactly what the predict and predict_proba functions do in this case, and given that the number of clusters is set to 3, the number of classes, the predict function will predict a label from $\{0, 1, 2\}$.

However, this still raises the question of how does the GaussianMixture assign particular labels to the clusters? In general, it is arbitrarily chosen, but in the example from sklearn that you linked, they cheat when initialising the cluster centers:

# Since we have class labels for the training data, we can
# initialize the GMM parameters in a supervised manner.
estimator.means_init = np.array([X_train[y_train == i].mean(axis=0)
                                for i in range(n_classes)])

The initial position for each cluster center is at the center of each class, which also has the additional consequence of correctly ordering the cluster labelling to match the original class labels. This means that the GMM predicting which cluster a new instance belongs to is equivalent to predicting which class it might belong to in this case. I believe this has been done for easy visualisation of the different covariance matrix options.

$\endgroup$
  • 2
    $\begingroup$ Note that only few unsupervised methods have such an option. K-means and GMM do. But for most, there is no natural way how to predict for unseen data. $\endgroup$ – Has QUIT--Anony-Mousse Feb 21 '19 at 6:58
  • $\begingroup$ @Anony-Mousse that makes sense. The whole idea of "cheating" during cluster initialization, and then use the clustering algorithm as a classifier still feels unorthodox to me. Is this a common practice? If not, are there supervised algorithms that works similar to GMM? $\endgroup$ – Chris Feb 21 '19 at 19:27
  • 1
    $\begingroup$ It is the other way around. You run a clustering algorithm and then use the resulting model for classification. Both k-means and GMM yield a simple nearest-neighbor type of classifier (with GMM using a Mahalanobis distance) as model. But it will classify into the clusters it found, not into the labels you also had. $\endgroup$ – Has QUIT--Anony-Mousse Feb 21 '19 at 23:10
  • $\begingroup$ @Anony-Mousse thanks! this is very informative $\endgroup$ – Chris Feb 22 '19 at 2:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.