1
$\begingroup$

I have implemented two type of clustering in python, using SciPy: one with hierarchical approach, and the other one with k-means.

In each cases I have used as input a two dimensional array X (documents, binary features) of 628 * 492.

#Hierarchical:

Z = linkage(X, 'ward')
clusters = fcluster(Z, max_k, criterion='maxclust').tolist()

#k-means:

clusters = kmeans2(X, k, iter_, minit="points")[1].tolist()

In the first case (hierarchical one) I got an array of size 628 (an array which the i-th element is the i-th document associate cluster)

In the second one I got an array of size 627. Why?

I got also this warning in the k-means execution (using different values for k):

/usr/lib/python3/dist-packages/scipy/cluster/vq.py:600: 
UserWarning: One of the clusters is empty. 
Re-run kmean with a different initialization.
warnings.warn("One of the clusters is empty. "

The cluster empty is the one associate to the missing document? If the document missing would be in the "middle" of the list this situation wouldn't led to incoherent results? (list indexing wrong)

$\endgroup$
  • $\begingroup$ The latter problem is likely because your data is not continuous but binary. Don't use k-means on such data, where the mean doesn't make sense. $\endgroup$ – Has QUIT--Anony-Mousse Sep 24 '16 at 16:30
  • $\begingroup$ Carefully check your clusters if they make any sense at all. $\endgroup$ – Has QUIT--Anony-Mousse Sep 24 '16 at 16:31
  • $\begingroup$ my clusters seems to be ok. They are somewhat similar to the ones which I got with the hierarchical approach $\endgroup$ – Nikaido Sep 24 '16 at 17:23
  • $\begingroup$ Do some blind testing. Get a cluster, and one that simply selects all documents with feature x = 1. Ask users to provide an explanation for the cluster, and ask them which cluster is better. Chances are, they will prefer the "cluster" that is a binary selector over the real clustering. $\endgroup$ – Has QUIT--Anony-Mousse Sep 24 '16 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.