1
$\begingroup$

I am doing KMeans clustering for sentence embeddings and my problem is the number of clusters. In general, feature size is an order of a few hundreds (in this case 768) and my concern is the sparsity of space. I tried to use gap statistic, but it just increases monotonically and has no maximum (I ended up with max 2048 clusters). Also, embeddings lie on a n-dimensional sphere rather than fill the space uniformly. My questions is: does it really make sense to use various "clustering metrics" to determine optimal number of cluster when the feature space is large?

$\endgroup$
0
$\begingroup$

There isn't a correct way to approach this problem. One common way is what you are doing, i.e. check for various values of $k$ and have a heuristic tell me the best value. Some such methods are are the elbow, silhouette and the gap statistic that you're using. Determining the number of clusters via such a method is perfectly valid; in fact that's what they're for.

Another approach would be to try hierarchical clustering on the data and see which level leads to the maximum difference in variance.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.