1
$\begingroup$

I am running k-mean clustering on ~200000 samples. The dataset has in total 14 features. One feature is id and the rest are categorical.

I have been playing with which features to include in the clustering and the metric Im using is Silhouette.

I would like advice on how to decide which analysis is better. A cluster with fewer features and a higher score (i.e., .8) or a cluster with more features and a lower score (i.e., 30)

My assumption is that the one with more features and a lower score is better because the algorithm has more information that describes the sample. However, those extra features may be making it harder for the algorithm to put the samples into groups.

Any advice/tips?

$\endgroup$
  • $\begingroup$ Do not include the id attribute!!! And reconsider your choice of methods - k-means is designed for continuous variables. So your results are highly questionable. Also, don't all clusters have the same features with k-means? $\endgroup$ – Has QUIT--Anony-Mousse Aug 27 '19 at 8:12
1
$\begingroup$

First of all, the Silhouette score comes always with visual inspection so be careful using it. The idea behind this score is having either Normally distributed clusters or having pretty compact clusters. If the intrinsic clusters within your data are not one of those, then this score is somehow meaningless.

Secondly, the question also needs a re-thinking. "Which features to include in the clustering?" ... well, all!

The characteristic of a dataset is more than just the combination of characteristics of each individual features. In these cases, usually Dimensionality Reduction algorithms are used, which reduce the dimensionality of data and extract/select informative features according to the entire feature set. Note that a feature that you might exclude, makes a meaningful partitioning in combination with some other features.

at the end I recommend having a look at K-Modes which is designed for clustering datasets with nominal attributes. Most probably helps you more than k-mans.

If you like to get some insight about Graph Clustering (which is also related to your problem and is one of typical solutions) you can comment here.

Good Luck!

$\endgroup$
  • $\begingroup$ Hi @kasra, thanks for the information! I would love to get more insight on graph clustering. Also, in some experiments, I left out a couple of features because one value in that feature dominated , so I was worried it would throw of the algorithm $\endgroup$ – Nelly Yuki Aug 27 '19 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.