0
$\begingroup$

I'm working on a project where I need to cluster data. After doing all the usual steps (in no distinct order: one-hot/BaseN encoding categorical data, doing a Quantile Transform due to none of the columns having any well-known distributions, scaling data, removing multicollinearity and outliers etc.), I found there's no discernable clusters in the data:

enter image description here

Doing DBSCAN with the ideal parameters yields a single cluster and abysmal silhouette scores, and it's the same if I run it on the whole dataset or the PCA version. (I also ran it with a bunch of random parameters to see if it improves anything, but nada.) Of course I could force K-Means to get more clusters, but the goal would be to get more distinct clusters.

My question is, is there a method for choosing features that would maximize the distance between clusters of the data? I can see something mildly akin to a cluster forming in this data, and I was wondering if there are any methods/metrics that I could use to isolate features that would result in more distinct clusters of the data (using every column available is not required by the business case for this, but trying everythin by hand is not viable due to the number of columns). Also, are there any other methods of preprocessing data that would exaggarate differences between clusters?

Thank you!

$\endgroup$
0
$\begingroup$

You may be able to find some non-linear embedding of your data which better clusters your data. This would assume that your data actually sits in some lower-dimensional space which linear methods like PCA are not powerful enough to recover. Sklearn.manifold would be a good place to start, e.g isomap or tSNE.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.