2
$\begingroup$

I have an already clustered data set (I wanna keep my x and y), where there's clearly a small group of elements in the middle that don't follow the expected patterns.

I can select them manually, but I wonder if there's a way of automating the selection part of these elements, efficiently.

Something like using just the grouping part of a clustering algorithm, I've been trying it with a threshold, but it doesn't produce good results in cases that won't form a circular cluster.

Already clustered data

$\endgroup$
0
$\begingroup$

It would be helpful to know which clustering technique are you using.

You can use

  • Partition-based Clustering: for example K-Means Clustering, not that good with outliers.
  • Hierarchical-based Clustering: Produces trees of clusters (Agglomerative, Divisive). You get a Dendogram.
  • Density-based Clustering: produces arbitrary shaped clusters, for example DBSCAN

If you are looking something other that a circular cluster and you need clusters within clusters, I would try DBSCAN. It locates regions of high density and separate outliers and it can find clusters within clusters.

If you are using Python you can use DBSCAN with sklearn

from sklearn.cluster import DBSCAN 

I hope that helps!

$\endgroup$
  • $\begingroup$ Can I keep my x and y with those algorithms? $\endgroup$ – Ângelo D Feb 11 at 11:37
  • $\begingroup$ If you use DBSCAN you would have to redo the analysis. Unless you apply the analysis to the clusters as x and y (not sure if you can do that). $\endgroup$ – daco Feb 11 at 11:54
  • $\begingroup$ But that is what I wanted, to keep the x and y that I have and try to group it. Because I need to use this algorithm I made. $\endgroup$ – Ângelo D Feb 11 at 12:04
0
$\begingroup$

You have it right, that you want your clustering to tell you which points are most anomalous. For k-means clustering it's the points that are farthest from their assigned cluster.

I don't see a reason to expect that the anomalies form a cluster themselves. If that's what you're expecting you may need to compute something else, like, a clustering of the points beyond a threshold?

Also consider a Gaussian mixture clustering, which is just like k-means except treats cluster assignments as soft and probabilistic. The outliers under that model might make more sense.

$\endgroup$
  • $\begingroup$ I get your point, but I don't want to calculate the clustering process again, I've tried using k-means and didn't get very good results. I just want something that I can apply after my clustering algorithm runs that can separate groups without changing x or y and doesn't need human suport. $\endgroup$ – Ângelo D Feb 11 at 17:47
  • $\begingroup$ Sure, why not cluster only the outliers? you're looking for a cluster of them, it seems. $\endgroup$ – Sean Owen Feb 11 at 21:16
  • $\begingroup$ No, I'm looking for a way to find the outliers automatically and not manually, something like you said "a clustering of the points beyond a threshold". $\endgroup$ – Ângelo D Feb 12 at 9:23
  • $\begingroup$ I just can't find a way of defining this threshold. $\endgroup$ – Ângelo D Feb 12 at 9:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.