0
$\begingroup$

So I used the elbow method to identify the optimal number of clusters, i.e. 4 in this case. After running K-means on dataset with 12 features, I get an array with the cluster number each row belongs to, like this:

Feature1    Feature2  .... Feature12   Clusters
<data>      <data>         <data>      2
<data>      <data>         <data>      0
<data>      <data>         <data>      0
<data>      <data>         <data>      3
<data>      <data>         <data>      1     and so on.

What should be the next step now? Should I individually check the differences between each cluster to identify the similarities and differences? If yes, how exactly?

$\endgroup$
1
$\begingroup$

You can run your K-Means for different values of K (e.g. k=2, 3, ..., N) and for each iteration you can compute the Silhouette metric, which provides a measurement on "how compact" your clusters are. You can plot your Silhouette score and find what is the K that makes the score minimum, or stable. Since this is a clustering operation and K-Means is a simple method (nothing bad about that), I would recommend using a second clustering method (e.g. t-SNE, DBSCAN) and apply a similar procedure to see whether there is some agreement between the models. This is a general framework in data science, just applied to a clustering problem.

$\endgroup$
0
$\begingroup$

What you did when fitting the k-means algorithm is you used your training data to determine the location of the 4 centroids.

To validate your results you can check if instances which have been classified as belonging to the same class make sense to be together intuitively. You can use some similarity metrics to get some statistics to back-up your claims. But training a k-means is usually meant for classifying new instances. You can do this by finding out which centroid a new instance is closest to.

$\endgroup$
0
$\begingroup$

I typically check the clusters using something like a pairplot. This generally gives a good idea of which features are driving the separation. But as far as I know, there's not a good hard-and-fast rule for determining if your clusters are "real" (i.e. meaningful). That's going to be up to you and your knowledge of the specific data you're working with.

As far as what to do after you determine your clusters are meaningful, it depends on what your original goal was. What data are you clustering and why? Without that information, it's hard to say what to do next.

$\endgroup$
0
$\begingroup$

You can calculate some metrics, which takes into consideration inter-distances in cluster and intra-distances between clusters. You can find more here:

https://en.wikipedia.org/wiki/Cluster_analysis#Internal_evaluation

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.