0
$\begingroup$

I'm currently in the middle of a clustering project and struggling to get acceptable results, is it commonplace that datasets just can't be clustered?

Context: I'm trying to cluster a relatively small dataset (c.10k observations) of mixed datatypes, mainly continuous, some categorical (label encoded). To give you an idea of the features, its things like customer engagement proxies, business size, revenue, tenure as a customer, amount of products purchased etc etc.

I have used many methods of scaling (minmax, z-scaling, powertransform, gmm etc.) and many different model types (kmeans, kproto, dbscan, agglom). I have achieved a silhouette score of approximately 0.35, which doesn't seem bad, but the clusters really don't seem to actually differentiate from each other!

Could it be a case that this problem cannot be solved with clustering? Or can data always be clustered and I just need to persevere?

$\endgroup$

1 Answer 1

1
$\begingroup$

Could it be a case that this problem cannot be solved with clustering? Or can data always be clustered and I just need to persevere?

Well, to understand that you should either visualize the data or known to which distribution they belong. Visualizing should be simpler, for example you can try t-SNE or UMAP to get a first understanding of how your data group in a high-dimensional (I suppose the dimensions to be greater than three) space. Unless you're given it, computing the data distribution can be quite difficult and also hard to interpret.

Consider that not all data cluster, for example uniform data do not show clusters and the same should be true for samples distributed around a single Gaussian. Clustering can handle multi-modal and complex shaped distributions, at least in principle.

I have used many methods of scaling.

Methods like k-means are affected by how you scale the data, so you expect to get different results according to the scaling and so you may want to use a clustering method that doesn't depend on the scaling.

I'm trying to cluster a relatively small dataset (c.10k observations) of mixed datatypes, mainly continuous, some categorical (label encoded).

I think trying to cluster mixed datatypes is your main issue. I believe you should separate continuous and categorical features, clustering them separately. The reason for this is that you're mixing radically different spaces which can complicate the clustering problem too much.

$\endgroup$
1
  • $\begingroup$ thanks for your response - my main question back is; if i'm clustering the two data types separately, how would you go about bringing the clusters together afterwards? $\endgroup$
    – roastbeeef
    Commented Jun 29, 2023 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.