Finding suitable measure for optimal number of clusters for the specified clustering method and specified data

Question

Suppose we have this kind of data and the preferred clusters (not in the case of the optimal number of clusters, but the shape of clusters) here:

I achieved the exact shape of clusters using KMeans! But I wonder which metric I should use to find the optimal number of clusters! I used silhouette coefficient, But it results in 2 clusters for these kinds of scattered data! So I'm slightly suspicious if I used a suitable metric for this problem.

To better explain this case, I want to dig into some details. One may ask what the x ticks and y ticks are.

• x ticks are the index of data! like [1, 2, ... length of data]
• y ticks shows the distance of each data point from a specific one.

So now it's a little bit clear what the idea is. I want data points within a distance range of (0, 🔴) to be in the same cluster, and distance range of (🔴, 🟡) in the same cluster, and so on. But the problem is I don't trust the silhouette coefficient because it always shows me 2 clusters through all varieties of data that I have in this specific problem!

Additional note:
the shape of clusters should be like the example (this is intended), no matter how scattered the data is.

Answer 1

https://www.scikit-yb.org/en/latest/api/cluster/elbow.html#:~:text=If%20the%20line%20chart%20looks,fits%20best%20at%20that%20point. I would recommend approaching the problem using the elbow method with the metric set as distortion ( by default) to find the optimal number of clusters.