trade offs between number of features with its score

Question

I am running k-mean clustering on ~200000 samples. The dataset has in total 14 features. One feature is id and the rest are categorical.

I have been playing with which features to include in the clustering and the metric Im using is Silhouette.

I would like advice on how to decide which analysis is better. A cluster with fewer features and a higher score (i.e., .8) or a cluster with more features and a lower score (i.e., 30)

My assumption is that the one with more features and a lower score is better because the algorithm has more information that describes the sample. However, those extra features may be making it harder for the algorithm to put the samples into groups.

Any advice/tips?

Answer 1

First of all, the Silhouette score comes always with visual inspection so be careful using it. The idea behind this score is having either Normally distributed clusters or having pretty compact clusters. If the intrinsic clusters within your data are not one of those, then this score is somehow meaningless.

Secondly, the question also needs a re-thinking. "Which features to include in the clustering?" ... well, all!

The characteristic of a dataset is more than just the combination of characteristics of each individual features. In these cases, usually Dimensionality Reduction algorithms are used, which reduce the dimensionality of data and extract/select informative features according to the entire feature set. Note that a feature that you might exclude, makes a meaningful partitioning in combination with some other features.

at the end I recommend having a look at K-Modes which is designed for clustering datasets with nominal attributes. Most probably helps you more than k-mans.

If you like to get some insight about Graph Clustering (which is also related to your problem and is one of typical solutions) you can comment here.

Good Luck!