I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is an outlier or not. What is the connection between percentile value and threshold and how can we decide if a point is an outlier or not using a threshold value?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
You have a distribution of data (multidimensional or not it does not matter). And anomalies will be scattered towards corners of this dataset. Hence low percentile will indicate that certain points are anomalies in the data---cornered in. Threshold is, in this context, just a cutt-off value when you claim something is anomaly.
answered Mar 10, 2020 at 14:56
-
$\begingroup$ Is there any upper or downrange from threshold to determine if a point is anomalous or not ? $\endgroup$ Commented Mar 10, 2020 at 15:23
-
$\begingroup$ It depends on the dataset. Generally the bigger the dataset, the bigger the cutoff, since its easier for points to land into certain extremes in different dimensions. $\endgroup$ Commented Mar 10, 2020 at 15:34
$\begingroup$
$\endgroup$
Usually outliers are considered to be normally distributed (although that may not be the case in real life). Therefore, the anomaly threshold is computed in terms of the standard deviation of the dataset. Common choices are 3$\sigma$ or 5$\sigma$, i.e., the data point is considered an outlier if it deviates from the mean by more than 3 or 5 $\sigma$. Under the assumption of normal distribution, those correspond to $p$-values of $3\times 10^{-3}$ or $3 \times 10^{-7}$. However, these are likely to be underestimates of the real $p$-values for a real (non-normal) distribution. So it's a good idea to check that your model is not producing too many false positive anomaly reports.
