I was asked to run a clustering analysis to assess the validity of labels for a manually labelled dataset.

I can simply save the actual labels (4 classes: 0, 1, 2, 3) and run clustering analysis (let's say python sklearn AgglomerativeClustering) on the rest of the data, specifying to get 4 clusters. Now I want to compare labels obtained from clustering and the original ones (manually labelled) to check if the manual labelling worked well.

The problem is that "cluster labels" have arbitrary values assigned by the clustering algorithm, for example, all samples labelled as 0 could have been labelled as 1, 2 or 3 and vice versa. They could also be A, B, C, D, or blue, green, red, yellow, they just need to be 4 distinct groups (clusters).

Now, how can I run the comparison? I would need a method to "convert" cluster label values to the original ones, hence cluster label 0 corresponds to original label 0? or original label 1 or 2 or 3? Once this issue is solved, I should be able to compare the two labels, count the number of matches between cluster and original labels, see how many samples were wrongly labelled, and share them with the experts.

I would like to use the AgglomerativeClustering algorithm for this task, but any other algorithm would be fine, even if I think that I would always face this issue, as clustering is an unsupervised learning method.


1 Answer 1

  1. You should not check whether an object belongs to a true cluster (as is done in the classification task), but count the number of pairs of objects that were correctly assigned to one cluster.
  2. In quality metrics (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html, for example), this is done automatically.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.