# How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of the feature vector for each sample is 40. Is there any clear way to compare the performance of unsupervised algorithms?

• @mikalai It is exactly what I have asked – Alireza Zolanvari Mar 22 '19 at 9:23

For unlabeled data-sets, unsupervised anomaly detectors can be compared either subjectively or objectively.

1. Subjective comparison: based on our domain-knowledge and by using some visualizations and statistics, we can compare two detectors and select the one that outputs better anomalies subjectively.

1. Here is a well-cited survey on unsupervised anomaly detectors that compares the algorithms on labeled data-sets (with known, domain-specific outliers) using AUC, and concludes that local detectors (such as LOF, COF, INFLO and LoOP) are not good candidates for global anomaly detection: 2016 A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data
2. Objective comparison: possible in theory, impossible in practice.

Requirements for objective comparison:

1. Anomaly definition: $$x$$ is an anomaly if $$P(x)< t$$ for some threshold $$t$$,

2. Anomaly detector requirement: $$D$$ is an anomaly detector if for every detected $$x$$, $$P(x)< t$$,

3. Comparing anomalies: $$x_1$$ is more anomalous than $$x_2$$ if $$P(x_1) or equivalently $$r(x_1, x_2) = P(x_1) / P(x_2) < 1$$,

4. Comparing anomaly detectors: proposal $$x_1$$ from detector $$D_1$$ is better than $$x_2$$ from $$D_2$$ if $$r(x_1, x_2) < 1$$,

As you can see, for qualification and comparison of two detectors we need to know the underlying $$P(x)$$ or at least $$r(x_1, x_2)$$. But if we know these quantities (which act as a judge $$J$$) or at least a close enough estimation of them, we have a better anomaly detector $$J$$ and can throw $$D_1$$ and $$D_2$$ away! We plug any observation $$x$$ or pair of observations $$x_1$$ and $$x_2$$ into $$J$$ and check which one is an anomaly or which one is more anomalous, done! So it is impossible to compare two anomaly detectors objectively unless we have a better anomaly detector (judge). So we should use a subjective comparison.

• Please check the question update. Each sample has about 40 features and subjective comparison is not very practical. – Alireza Zolanvari Mar 20 '19 at 11:04