0
$\begingroup$

I have a dataset that I am analysing to find the optimal number of clusters using k-means.

I am testing the number of clusters from [1..11] - which produces the following plot:

enter image description here

The original dataset has six classes but the elbow plot shows the bend really occurring at 3 clusters. For curiosity I overlaid a line on the plot from 11 clusters and back and it is almost a straight line to 6 clusters - which indicates to me that the real elbow is at 6, but it is subtle to see.

So, visually 3 looks to be the right answer, but given the known number of classes (6) the straight line I drew indicates 6...

Question:

  • How should you correctly interpret an elbow plot like this (especially when you are not given the classes)?
  • Would you say the elbow is at 3 or 6?
$\endgroup$
2
  • 2
    $\begingroup$ Using the Elbow method to determine the no of clusters is not a preferred way as there is usually no distinctive "knee" in the plot. If you have some previous knowledge about the data (somewhat similar to the idea of semi-supervised learning), then you may use that to determine the no of clusters. As you already know there are six classes in the dataset, then you should use 6 clusters, not 3. $\endgroup$
    – Imran
    Apr 17, 2022 at 8:54
  • $\begingroup$ Thanks Imran. "As you already know there are six classes in the dataset, then you should use 6 clusters, not 3" Fair enough, but what I have above is just an example to illustrate the point of the question. The question is "If I want to use the elbow method and I do not know the number of clusters, what is the most correct way to find the elbow?" $\endgroup$
    – Bryon
    Apr 17, 2022 at 8:59

2 Answers 2

0
$\begingroup$

As your question implies, it can be difficult to determine the number of clusters from an elbow plot as it is a heuristic approach rather than an exact method.

In the past I have used Cluster Evaluation By Prediction Strength with some success. There is an accessible intro to this in this blog post, along with Python code. Some R code is available at this github. I have also written some R code for my personal use in the one dimensional case which is available in one of my blog posts here.

$\endgroup$
0
$\begingroup$

The correct way to find the optimal clusters depends on a couple of things. your reasearch question, proportion of the instances in the cluster, elbow method, you can also try silhouette method. You can also say 6 clusters from your graph because the graphs is still droppping at 3. Relatively, it is straightening out at 6. It all depends on what exactly you want to explore.!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.