1
$\begingroup$

I'm working on an anomaly detection development in Python.
More in details, I need to analysed timeseries in order to check if anomalies are present.
An anomalous value is typically a peak, so a value very high or very low compared to other values.

The main idea is to predict timeseries values and, using thresholds, detect anomalies.

Thresholds are calculated using the error, that is the real values minus the predicted ones.
Then, mean and standard deviation of the error are performed.

The upper threshold is equals to mean + (5 * standard deviation).
The lower threshold is equals to mean - (5 * standard deviation).

If the error exceeds thresholds is marked as anomalous.

What doesn't work with this approach is that if I have more than one anomalous value in a day, they are not detected. This because error, mean and standard deviation are too much influenced by anomalous values.

How can i fix this problem? Is there another method that i can use to identify thresholds without this issue?

Thank you

$\endgroup$
  • $\begingroup$ Does your data vary significantly from day to day? $\endgroup$ – Akavall Oct 29 '19 at 22:24
  • $\begingroup$ no, data are quite similar $\endgroup$ – Giordano Oct 30 '19 at 8:00
  • $\begingroup$ In the equation: The upper threshold is equals to mean + (5 * standard deviation) are mean and standard deviation of the one day sample or is it of a much larger period? $\endgroup$ – Akavall Oct 30 '19 at 19:40
4
$\begingroup$

Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard deviation formula.

You will probably have to change your critical value to something other than 5 to get the same kind of coverage. According to Wikipedia, you'll want the new critical value to be $5\sqrt{\frac{\pi}{2}}$ if your data are iid Gaussian.

An alternative that might be more difficult to implement, but is probably more statistically appropriate, is to use trimmed estimators for the mean and standard deviation. With trimmed estimators, you throw away the most extreme values in your data (the proportion of which is specified beforehand), and estimate your statistics on the remaining data.

The estimator for the mean would be the truncated mean, and the Wikipedia page for trimmed estimators mentions how to get a decent estimator for the standard deviation from the interquartile range.

I hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.