1
$\begingroup$

I have a quite basic question: A standard deviation is defined such that around ~66 % of the data lies within it. And around ~99 % within three standard deviations.

When I wanna' use the standard deviation as an outlier detection, I struggle with this definition as there will always be outlier. But I probably have a misunderstanding somewhere, somehow..

In other words: 3 standard deviations contain always about 99 % of the data, hence there will always be $ outlier > 3 std $, no matter if they are outlier.

What is the issue I don't see?

|improve this question|||||
$\endgroup$
  • $\begingroup$ The metrics you site assume normal distribution; it is not true for any distribution. $\endgroup$ – Akavall Sep 13 '19 at 13:30
  • $\begingroup$ wikipedia doesn't specifiy it has to be used for the standard distribution(?) $\endgroup$ – Ben Sep 13 '19 at 14:17
1
$\begingroup$

The standard deviation method is skewed by the presence of outliers.

One of the more robust methods which is reasonably simple to implement is Tukey fences (Wikipedia) which relies on quartiles and medians.

|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ Thanks for making me aware of that! $\endgroup$ – Ben Feb 27 at 10:36
1
$\begingroup$

Standard deviation isn't an outlier detector. It can't tell you if you have outliers or not.

What it will do is effectively remove outliers that do exist, with the risk of deleting a small amount of inlying data if it turns out there weren't any outliers after all. As such, I think it's useful as a "quick-and-dirty don't want to spend too much time on this problem" method of ensuring your dataset only contains inlying data points. As long as you understand the risk of potentially binning 1% of inlying datapoints, I think it's fine.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks, I don't get what you mean by "binning". I don't use a histogram here and this is the only situation or object I'm used to this term? Despite from that: A distance measure is not an outlier detection as well, but you can efficiently use it for this purpose. $\endgroup$ – Ben Sep 13 '19 at 12:22
  • 1
    $\begingroup$ @Ben my apologies; the word "binning" is an idiom in England for "getting rid of". I mean that by simply removing any data samples lying outside the range mean() +/- 3 * standard_deviation() you can quickly remove any outliers that do exist, but will delete 1% of your inlying data if it turns out there are no outliers. $\endgroup$ – Dan Scally Sep 13 '19 at 12:25
  • $\begingroup$ This implies you define an outlier as > 3 std. In case there are none, then I wouldn't delete any data points?! $\endgroup$ – Ben Sep 13 '19 at 14:20
0
$\begingroup$

I think at the core you might have understood the purpose of Outlier detection. You essentially balance two things: detect all outliers/ anamolies, have as little as possible detection of outliers/anamolies that are "just" random noise.

Depending on for what you need it it might be that you want to detect all outliers (if there impact is very negative for example) -- even at the cost of false alarms.

But maybe "small" outliers are fine for you and you want only to detect the really big deviations. This is what you can use your standard deviations for.

In the end you need to optimize the "cost" of the expected false positives and false negatives for your specific scenario. A classical example for this is Fraud detection: You can avoid all Fraud by rejecting all Customers or you can avoid any proper Customer being rejected by allowing all Fraudsters - for most businesses the truth lies somewhere in the middle -- so they essentially tune the parameters of their outlier detection by comparing the costs of added additional Fraudsters with the lost Revenue of additional rejected proper Customers.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks, the question was rather: When I use as threshold, e.g., 3 standard deviations, there will always be outlier. So what is a proper threshold then? $\endgroup$ – Ben Sep 13 '19 at 12:23
  • $\begingroup$ Thats what I tried to explain - it depends what you want to achieve. If you just want to make statistical noise more unlikely go to five sigma - but you will have the same problem (only way less often) $\endgroup$ – El Burro Sep 13 '19 at 13:03
  • $\begingroup$ I want to detect outliers :) $\endgroup$ – Ben Sep 13 '19 at 14:18
  • $\begingroup$ But outliers you expect based on the distribution of your data or unexpected ones? $\endgroup$ – El Burro Sep 17 '19 at 8:09
  • $\begingroup$ I think both - it is meant for an anomaly detection for a technical system $\endgroup$ – Ben Sep 17 '19 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.