How to run hdbscan clustering faster?

Question

I'm using hdbscan to cluster embedding output from BERT, which took in a data file of >150k chat messages. The embedding process took a little over 4 minutes, but as of this writing the hdbscan clustering process has taken > 1 hour and there's no end in sight.

What also seems weird to me is when I look at the system resource consumption (I'm on Windows 10), the script I'm running is only taking up 7% of CPU (i9-10850K) and 1.2GB of memory (out of 128GB). So I have a few questions,

1. Why is this not consuming more system resources?
2. Is there some way for me to parallelize this?

Lastly here's the code FYI.

data = parseMessages('raw-data-150k-new-delim.txt', 'ð')
model = SentenceTransformer('paraphrase-MiniLM-L6-v2')

embeddings = model.encode(data, show_progress_bar=True, convert_to_tensor=True)

clusters = hdbscan.HDBSCAN(
    min_cluster_size=15,
    min_samples=15,
    metric='euclidean',
    cluster_selection_method='eom').fit(embeddings)

Thanks!!

Answer 1

hdbscan greatly prefers lower dimensional data than the output of sentence-BERT. Ultimately the hdbscan library wants to use KDTrees of BallTrees for efficient nearest neighbor querying, and these work best in 50 dimensions or less. With higher dimensional data the library defaults to using a much slower and far more expensive code path. To make this run faster the best approach is going to be to reduce the dimensionality of the data. In my (high biased) opinion UMAP will be a good way to achieve this, but you could also try PCA, or other dimension reduction techniques.