how to compare between kmeans and hierarchical clustering results

Question

I am using 2 types of clustering algorithm I apply hierarchical clustering the K-means clustering using python sklearn library

Now the results are a little bit different so how can I compare the results and which algorithm to use? because I want to write a conclusion for a set of unlabeled data.

Is there any benefit to use multiple algorithms and compare between them?

Answer 1

In general, even using k-means more than once will create slightly different clusters (that s if you do not set a random seed).

Ideally, the profiles of the clusters and the distribution of data points into them will be the similar. If this is the case then it shouldn't really matter which one you choose. If not then, I suggest you examine carefully what caused any major differences.

If you really want to chose, then:

• you can use a metric like total within distance or silhouette (ex. select the clustering solution with the lowest total with distance or with maximum average silhouette),
• you can use your judgement to select the clustering solution that makes more sense business wise.

As mentioned before, ideally, metrics like total within distance or silhouette should not differ much. Hence, it all comes down to using the second criterion.

As for the part of the question about any benefits from using multiple algorithms,

• if you are referring to k-means and hierarchical clustering, you could first perform hierarchical clustering and use it to decide the number of clusters and then perform k-means. This is usually in the situation where the dataset is too big for hierarchical clustering in which case the first step is executed on a subset.
• in general, since not all clustering algorithms are suitable for every case it is useful to use multiple algorithms. For example k-means and hierarchical clustering are good at detecting clusters that have a spherical/ball shape. They would perform poor on more complex shaped data. A problem that DBSCAN does not have (though DBSCAN has other disadvantages, see discussion at Wikipedia)