1
$\begingroup$

I'm trying to determine number of clusters for k-means using sklearn.metrics.silhouette_score. I have computed it for range(2,50) clusters. How to interpret this? What number of clusters should I choose?

enter image description here

$\endgroup$
2
$\begingroup$

They are all bad. A good Silhouette would be 0.7

Try other clustering algorithms instead.

$\endgroup$
5
  • $\begingroup$ I'm clustering text data. So, I can compute the most popular word in each cluster. When I see at top_n words, I can conclude that algorishm works well. But I don't know what k is the best number of clusters. I don't understand why this metric shows so bad result. $\endgroup$ – Толкачёв Иван Oct 8 '16 at 22:49
  • $\begingroup$ Pick 10 random documents. Assign every document to the most similar of them. The top_n words will usually still look good... $\endgroup$ – Has QUIT--Anony-Mousse Oct 9 '16 at 0:07
  • $\begingroup$ But I can look at documents and almost all documents corespond to this words. $\endgroup$ – Толкачёв Иван Oct 9 '16 at 9:43
  • $\begingroup$ Yes, but so would they in the random example I gave. A more relevant evaluation is word coherence. see "Reading Tea Leaves: How Humans Interpret Topic Models", papers.nips.cc/paper/… $\endgroup$ – Has QUIT--Anony-Mousse Oct 9 '16 at 10:14
  • $\begingroup$ Sorry, I looked at the words that correspond to the center of cluster. $\endgroup$ – Толкачёв Иван Oct 9 '16 at 10:38
0
$\begingroup$

Silhouette measures BOTH the separation between clusters AND cohesion in respective clusters.

Intuitively speaking, it is the difference between separation B (average distance between each point and all points of its nearest cluster) and cohesion A (average distance between each point and all other points in its cluster) divided by max(A,B).

It is a value between -1 and 1, the higher the better (negative value means that the point is more closer to the nearest cluster than to its own, which is quite a problem).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.