I have a dataset (all numerical) of 50K records containing 500 features. we are trying to find fingerprints. Meaning that we would like to cluster the data and report one of the nodes in each cluster as representative of that cluster (meaning that each node in that cluster is mostly similar to that representative compared to any other node in other clusters). So we won't have any noise (meaning that all nodes should be represented). I have done K-means, Kmedoids, hierarchical, dbscan, hdbscan, etc. Each having their own issues.
Kmeans doesn't report one of the nodes as center for the cluster but the center point which might not be one of our nodes. So we switched to Kmedoids. which the results are too dependent of the initial seed. Also not knowing K is another problem.
Then we used elbow method to find K. But since not sure what is the upper limit for K, the plot is not very Elbow-y but more like a gradual decrease. Then we tried Sillouette0score method and I got WAY too big number for clusters (something over 200 which doesn't seem right to the field expert). Same problem with the number for K generated by Affinity Propagation (too big of K).
Since we have tens of thousands of data, Meanshift doesn't relly work properly. Also HDBSCAN reports 264 labels which again seems unreasonable.
I would like to try some dimensionality reduction methods for this data. But not sure what would work well. tSNE is too focused on visualization which doesn't seem best fit for our use. Any advice would be VERY appreciated.