Notion of cluster centers and cluster comparison in Density Based Algorithms

Question

I have done some research on clustering algorithms since for my goal is to cluster noisy data and identify outliers or small clusters as anomalies. I consider my data noisy because of my main feautures can have quite varying values. Therefore, my focus has been on density based algorithms with quite some success.

However, I am unable to grasp the idea of cluster comparison in such algorithms since the notion of cluster centers cannot be properly defined.

My dataset constists of network flows and I split the dataset in subsets based on an identifier. After applying clustering on each subset I want to be able to compare the clusters that are created on each subset so that I can compare the subsets themselves in some context.

Would appreciate some help from data scientist gurus on how to approach the concept of cluster comparison or cluster center in such algorithms.

Thanks all!

Answer 1

You can either use the medoid, you can sometimes compute a centroid (and just ignore that it may be outside of the cluster), or you can do pairwise comparisons and take the average of that rather than comparing centers.

Answer 2

You could employ Gaussian Mixture Modeling (or a variant). The objective is to fit a Gaussian kernel $N(\mu, \sigma)$ to each of your sub-clusters. Baseline distance measure between pair of sub-clusters, you are seeking, could be $L_2$ norm of their means $d(\mu_1, \mu_2)$. The sub-clusters would typically have different standard deviations. You can factor that into your distance measure, to improve interpretation of distance measure. You can use this to identify outliers. Typically, outliers would be characterized by joint criteria of highest mean distance from all other sub-clusters, and a low variance.