HDBSCAN Outlier Detection and labeling

Question

I am using the HDBSCAN algorithm so as to perform unsupervised clustering and detect outliers. Based on the documentation there are two outputs from the clustering process that can give insight on which points are outliers.

1. The GLOSH outlier detection algorithm that gives a degree of certainty of whether a point is an outlier or not.
2. The HDBSCAN labels that if an element in not part of a cluster is considered as noise and has the corresponding label.

I have been working on some data, and I have noticed that these two approaches do not give the same results. That means that there are points in my dataset that are labeled as not part of a cluster but are not detected by the GLOSH outlier algorithm. Should I consider the union of these two approaches as my outliers or I am missing something in their interpretation?

Answer 1

GLOSH works with regard to local outliers, thus if a region is very dense and then has a few sparse points near it GLOSH will rate those sparse points as more outlying than other noise points in sparser regions. This is because it is considered particularly odd for a point to be near, but not in a very dense region -- local to the region the point is a significant outlier.

With regard to your question: if you care about local outliers (things that are odd relative to the density of the nearby region) then GLOSH is a good choice. This would be useful, for example, in detecting network intrusions since an attacker will be trying to behave like normal users, but will be slightly outside the dense normal user cluster. If, on the other hand, you are more interested in generally outlying points then merely selecting the noise points from the clustering will pick out points in sparser regions of the space.

Finally note that selecting noise points gives you only a single threshold; in contract GLOSH provides a range of scores and you can tune the threshold to find the number of outliers you desire.