Determining the optimal number of clusters by elbow method

Question

I have a dataset that consists of 700 categorical columns and around 6000 rows. I created 2-50 clusters with the k-mode algorithm and plotted the cost function to determine the optimal number of clusters.

This is what the plot looks like

I am unsure how determine what is the optimal number of clusters. The cost function seems to converge at 48 clusters, which seems alot considering i have only 700 categorical columns. On the other hand at 24 clusters the curve seems to be less steep.

Could someone shed some light into this, how to analyse the plot correctly?

Answer 1

That plot looks weird. Please take a look at the link below.

https://towardsdatascience.com/cheat-sheet-to-implementing-7-methods-for-selecting-optimal-number-of-clusters-in-python-898241e1d6ad

Also, look at Affinity Propagation, which selects the optimal number of clusters for you, automatically.

https://scikit-learn.org/stable/auto_examples/cluster/plot_affinity_propagation.html#sphx-glr-auto-examples-cluster-plot-affinity-propagation-py

Answer 2

Using the elbow method, you can determine the number of clusters quantitatively in an automatic way (as opposed to doing it by eye using this method), if you introduce the quantity called the "elbow strength". Basically, it is based on the derivative of the elbow-plot with some more information-enhancing tricks. More details about the elbow strength can be found in the supplementary information of the following publication:

https://iopscience.iop.org/article/10.1088/2632-2153/abd87c

Alternatively, you can also try the silhouette method:

https://en.wikipedia.org/wiki/Silhouette_(clustering)