How to compare two unsupervised anomaly detection algorithms on the same data-set?

Question

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of the feature vector for each sample is 40. Is there any clear way to compare the performance of unsupervised algorithms?

Answer 1

For unlabeled data-sets, unsupervised anomaly detectors can be compared either subjectively or objectively.

1. Subjective comparison: based on our domain-knowledge and by using some visualizations and statistics, we can compare two detectors and select the one that outputs better anomalies subjectively.

   1. Here is a well-cited survey on unsupervised anomaly detectors that compares the algorithms on labeled data-sets (with known, domain-specific outliers) using AUC, and concludes that local detectors (such as LOF, COF, INFLO and LoOP) are not good candidates for global anomaly detection: 2016 A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data

2. Objective comparison: possible in theory, impossible in practice.

Requirements for objective comparison:

1. Anomaly definition: $x$ is an anomaly if $P(x)< t$ for some threshold $t$,

2. Anomaly detector requirement: $D$ is an anomaly detector if for every detected $x$, $P(x)< t$,

3. Comparing anomalies: $x_1$ is more anomalous than $x_2$ if $P(x_1)<P(x_2)$ or equivalently $r(x_1, x_2) = P(x_1) / P(x_2) < 1$,

4. Comparing anomaly detectors: proposal $x_1$ from detector $D_1$ is better than $x_2$ from $D_2$ if $r(x_1, x_2) < 1$,

As you can see, for qualification and comparison of two detectors we need to know the underlying $P(x)$ or at least $r(x_1, x_2)$. But if we know these quantities (which act as a judge $J$) or at least a close enough estimation of them, we have a better anomaly detector $J$ and can throw $D_1$ and $D_2$ away! We plug any observation $x$ or pair of observations $x_1$ and $x_2$ into $J$ and check which one is an anomaly or which one is more anomalous, done! So it is impossible to compare two anomaly detectors objectively unless we have a better anomaly detector (judge). So we should use a subjective comparison.