0
$\begingroup$

Apart from ELBOW rule and silhoutte coefficient is there any other better methods to pick correct number of clusters in recent years ?

$\endgroup$
3
  • $\begingroup$ datascience.stackexchange.com/a/78448/47376 This might answer your question $\endgroup$ Jun 29, 2021 at 14:24
  • $\begingroup$ see also: datascience.stackexchange.com/a/87174/100269 $\endgroup$
    – Nikos M.
    Jun 30, 2021 at 13:03
  • $\begingroup$ thanks for the links. In various occasions, especially while dealing with big data I face issues when we iterate the clustering metrics for various number of clusters. Is there any sampling based approach or some approximation based approach that can scale well with the data ? $\endgroup$
    – yash kumar
    Jul 1, 2021 at 12:24

1 Answer 1

1
$\begingroup$

There is a metric named Davies-Bouldin Index

Just like silhouette score, It can be used when cluster labels are not known.

From Scikit-learn documentation:

This index signifies the average similarity between clusters, where the similarity is a measure that compares the distance between clusters with the size of the clusters themselves.

Zero is the lowest possible score. Values closer to zero indicate a better partition.

Advantages: The computation of Davies-Bouldin is simpler than that of Silhouette scores. The index is computed only quantities and features inherent to the dataset.

Drawbacks: The Davies-Boulding index is generally higher for convex clusters than other concepts of clusters, such as density based clusters like those obtained from DBSCAN. The usage of centroid distance limits the distance metric to Euclidean space.

In order to find a suitable number of clusters, you can vary the number of clusters and calculate the index.

$\endgroup$
2
  • $\begingroup$ That's great. But usually in real world scenario we deal with huge number of features that questions the usage of euclidean distance. What would be your suggestion on that ? $\endgroup$
    – yash kumar
    Jul 1, 2021 at 12:22
  • $\begingroup$ I do not know and exact method for the propose you mention, but, If X dimension is a concern, I would go with an unsupervised dimensionally reduction method, such as PCA or Feature agglomeration prior clustering, another options I can think of are using variance threshold or/and removing monotonic correlated (Spearman) features $\endgroup$
    – Multivac
    Jul 1, 2021 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.