6
$\begingroup$

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of the feature vector for each sample is 40. Is there any clear way to compare the performance of unsupervised algorithms?

$\endgroup$
  • $\begingroup$ @mikalai It is exactly what I have asked $\endgroup$ – Alireza Zolanvari Mar 22 '19 at 9:23
2
$\begingroup$

For unlabeled data-sets, unsupervised anomaly detectors can be compared either subjectively or objectively.

  1. Subjective comparison: based on our domain-knowledge and by using some visualizations and statistics, we can compare two detectors and select the one that outputs better anomalies subjectively.

    1. Here is a well-cited survey on unsupervised anomaly detectors that compares the algorithms on labeled data-sets (with known, domain-specific outliers) using AUC, and concludes that local detectors (such as LOF, COF, INFLO and LoOP) are not good candidates for global anomaly detection: 2016 A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data
  2. Objective comparison: possible in theory, impossible in practice.

Requirements for objective comparison:

  1. Anomaly definition: $x$ is an anomaly if $P(x)< t$ for some threshold $t$,

  2. Anomaly detector requirement: $D$ is an anomaly detector if for every detected $x$, $P(x)< t$,

  3. Comparing anomalies: $x_1$ is more anomalous than $x_2$ if $P(x_1)<P(x_2)$ or equivalently $r(x_1, x_2) = P(x_1) / P(x_2) < 1$,

  4. Comparing anomaly detectors: proposal $x_1$ from detector $D_1$ is better than $x_2$ from $D_2$ if $r(x_1, x_2) < 1$,

As you can see, for qualification and comparison of two detectors we need to know the underlying $P(x)$ or at least $r(x_1, x_2)$. But if we know these quantities (which act as a judge $J$) or at least a close enough estimation of them, we have a better anomaly detector $J$ and can throw $D_1$ and $D_2$ away! We plug any observation $x$ or pair of observations $x_1$ and $x_2$ into $J$ and check which one is an anomaly or which one is more anomalous, done! So it is impossible to compare two anomaly detectors objectively unless we have a better anomaly detector (judge). So we should use a subjective comparison.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Please check the question update. Each sample has about 40 features and subjective comparison is not very practical. $\endgroup$ – Alireza Zolanvari Mar 20 '19 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.