I have a problem that is about identifying clusters of highly correlated items. I initially focused on building a model and features that put similar data items close to each other. The main challenge is that I have a case of imbalanced data, as follows:
- Tens of Millions of items are random and not necessarily correlated.
- Hundreds of clusters of items (composed of 10-1000s of elements) exist* or may emerge. *I do have partial ground truth for the existing ones.
- Clusters are very different, in size and properties.
I'd like to return the identified clusters, and the elements within each cluster. F1 should be a good measure.
To move forward, I can think of threshold-based hierarchical clustering. Are there other techniques to consider?