1
$\begingroup$

I have done some research on clustering algorithms since for my goal is to cluster noisy data and identify outliers or small clusters as anomalies. I consider my data noisy because of my main feautures can have quite varying values. Therefore, my focus has been on density based algorithms with quite some success.

However, I am unable to grasp the idea of cluster comparison in such algorithms since the notion of cluster centers cannot be properly defined.

My dataset constists of network flows and I split the dataset in subsets based on an identifier. After applying clustering on each subset I want to be able to compare the clusters that are created on each subset so that I can compare the subsets themselves in some context.

Would appreciate some help from data scientist gurus on how to approach the concept of cluster comparison or cluster center in such algorithms.

Thanks all!

$\endgroup$
1
$\begingroup$

You can either use the medoid, you can sometimes compute a centroid (and just ignore that it may be outside of the cluster), or you can do pairwise comparisons and take the average of that rather than comparing centers.

$\endgroup$
  • $\begingroup$ Thank you for your answer, indeed after a bit of research medoid is what made the most sense for me. When you refer to pairwise comparison can you give me an example because it can be the case that the clusters don't have the same size. $\endgroup$ – MikEKOU Jul 17 '17 at 19:14
  • $\begingroup$ Compute all N x M distances, take the average. $\endgroup$ – Anony-Mousse Jul 17 '17 at 19:50
  • $\begingroup$ I see, that does sound computation heavy though. Anyway thanks for your answers regarding the topic. $\endgroup$ – MikEKOU Jul 17 '17 at 20:00
  • $\begingroup$ The medoid is similarly expensive to compute. $\endgroup$ – Anony-Mousse Jul 18 '17 at 6:44
0
$\begingroup$

You could employ Gaussian Mixture Modeling (or a variant). The objective is to fit a Gaussian kernel N(mu, sigma) to each of your subclusters. Baseline distance measure between pair of subclusters, you are seeking, could be L2 norm of their means d(mu1, mu2). The subclusters would typically have different standard deviations. You can factor that into your distance measure, to improve interpretation of distance measure. You can use this to identify outliers. Typically, outliers would be characterized by joint criteria of highest mean distance from all other subclusters, and a low variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.