I've read a number of papers where the authors talk about "Unsupervised Hierarchical Agglomerative Clustering". They seem to imply that the algorithm determines the number of clusters based on a hyper-parameter:
We define the hetereogeneity metric within a cluster to be the average of all-pair jaccard distances, and at each step merge two clusters if the heterogeneity of the resultant cluster is below a specified threshold
When I search for python implementations of Agglomerative Clustering I keep coming up with sklearn, which requires the number of clusters to be specified aprior. In most examples this is computed by plotting a dendogram and then determining by what appears to be eyeballing the chart how many clusters - for example https://towardsdatascience.com/machine-learning-algorithms-part-12-hierarchical-agglomerative-clustering-example-in-python-1e18e0075019 I'd argue it's impossible from the chart alone to determine if 3 or 5 is the optimal (based on largest vertical distance). I believe this is Wards method but I'm not sure it's the same as ""merging clusters where the heterogeneity is below a threshold" and
Is this possible in sklearn, or is there another python implementation which does this? I feel at the very least there should be a way to process the dendogram programmatically rather than plotting it?
