I have a question on K-means algorithm about the choice of the k value. I read to choose the correct value of k, there are 2 methods:
Or the k value, can be chosen empirically without any methods?
For example, is it correct, to state k=2 since only two clusters were possible, either malignant or non-malignant?
Remember, k-means is an exploratory method, so there is no 'correct' answer in all cases for how many clusters there should be. the elbow method might say 5 clusters is best, but you can choose any other number if you fancy. You might, for example want to choose the smallest number of clusters that achieves a minimum performance. It depends on why you're doing the clustering.
Your example doesn't make sense without more details, can you edit your post to add more detail?
Using the elbow method, you can determine the number of clusters quantitatively in an automatic way (as opposed to doing it by eye using this method), if you introduce the quantity called the "elbow strength". Basically, it is based on the derivative of the elbow-plot with some more information-enhancing tricks. More details about the elbow strength can be found in the supplementary information of the following publication:
https://iopscience.iop.org/article/10.1088/2632-2153/abd87c
Alternatively, you can also try the silhouette method:
It happens all the time that business is looking for some specific clusters and any method like Elbow method does not align with the business objective. In that case we need to break the cluster or group the clusters. So say for example you want malignant & non - malignant and k=2 may not give you good cluster , you can create k=4 and analyze the cluster to assign them to category defined by you.
So, in short you can choose k of your choice but that may or may not give you lowest error or good cluster
