1
$\begingroup$

I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been looking at other clustering such as DBSCAN, which didn't do as well either, even after tuning the parameters.

I may have missed some steps or dimension reduction steps for kmeans, in which I am happy to go back to and see how it would be any different, but can anyone recommend any clustering algorithm or direct me anywhere that I can look into further?

I will have a look at agglomerative hierarchical clustering.

$\endgroup$
1
  • 3
    $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Sep 5, 2022 at 18:58

1 Answer 1

0
$\begingroup$

From sklearns clustering documentation

Inertia is not a normalized metric: we just know that lower values are better and zero is optimal. But in very high-dimensional spaces, Euclidean distances tend to become inflated (this is an instance of the so-called “curse of dimensionality”). Running a dimensionality reduction algorithm such as Principal component analysis (PCA) prior to k-means clustering can alleviate this problem and speed up the computations.

Although PCA is advised, I would look into truncated SVD as it is supposed to work better on sparse data.

Use the explained_variance_ratio output to work out what cummulative explained variance is suitable for you (ie 95%, 99%). Hopefully you can reduce the features to ~50 although there is no hard rule about this.

Lastly I guess we should ask do you know this data has cluster ground truth?

$\endgroup$
1
  • $\begingroup$ Thanks, I have used the Truncated SVD, and it seems to have done okay. I have also used Truncated SVD for agglomerative HCL, and it's giving similar results. I will use K-means++ to help with the initalisations of the centroid. And yes, I have the cluster ground truths to use, but I shall also look into a metric to test this. $\endgroup$
    – Is land
    Commented Sep 6, 2022 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.