I'm trying to determine number of clusters for k-means using sklearn.metrics.silhouette_score. I have computed it for range(2,50) clusters. How to interpret this? What number of clusters should I choose?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
They are all bad. A good Silhouette would be 0.7
Try other clustering algorithms instead.
answered Oct 6, 2016 at 23:45
-
$\begingroup$ I'm clustering text data. So, I can compute the most popular word in each cluster. When I see at top_n words, I can conclude that algorishm works well. But I don't know what
kis the best number of clusters. I don't understand why this metric shows so bad result. $\endgroup$ Commented Oct 8, 2016 at 22:49 -
$\begingroup$ Pick 10 random documents. Assign every document to the most similar of them. The top_n words will usually still look good... $\endgroup$ Commented Oct 9, 2016 at 0:07
-
$\begingroup$ But I can look at documents and almost all documents corespond to this words. $\endgroup$ Commented Oct 9, 2016 at 9:43
-
$\begingroup$ Yes, but so would they in the random example I gave. A more relevant evaluation is word coherence. see "Reading Tea Leaves: How Humans Interpret Topic Models", papers.nips.cc/paper/… $\endgroup$ Commented Oct 9, 2016 at 10:14
-
$\begingroup$ Sorry, I looked at the words that correspond to the center of cluster. $\endgroup$ Commented Oct 9, 2016 at 10:38
$\begingroup$
$\endgroup$
Silhouette measures BOTH the separation between clusters AND cohesion in respective clusters.
Intuitively speaking, it is the difference between separation B (average distance between each point and all points of its nearest cluster) and cohesion A (average distance between each point and all other points in its cluster) divided by max(A,B).
It is a value between -1 and 1, the higher the better (negative value means that the point is more closer to the nearest cluster than to its own, which is quite a problem).
