I am doing KMeans clustering for sentence embeddings and my problem is the number of clusters. In general, feature size is an order of a few hundreds (in this case 768) and my concern is the sparsity of space. I tried to use gap statistic, but it just increases monotonically and has no maximum (I ended up with max 2048 clusters). Also, embeddings lie on a n-dimensional sphere rather than fill the space uniformly. My questions is: does it really make sense to use various "clustering metrics" to determine optimal number of cluster when the feature space is large?
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
There isn't a correct way to approach this problem. One common way is what you are doing, i.e. check for various values of $k$ and have a heuristic tell me the best value. Some such methods are are the elbow, silhouette and the gap statistic that you're using. Determining the number of clusters via such a method is perfectly valid; in fact that's what they're for.
Another approach would be to try hierarchical clustering on the data and see which level leads to the maximum difference in variance.
