I'm working on an anomaly detection development in Python.
More in details, I need to analysed timeseries in order to check if anomalies are present.
An anomalous value is typically a peak, so a value very high or very low compared to other values.

The main idea is to predict timeseries values and, using thresholds, detect anomalies.

Thresholds are calculated using the error, that is the real values minus the predicted ones.
Then, mean and standard deviation of the error are performed.

The upper threshold is equals to mean + (5 * standard deviation).
The lower threshold is equals to mean - (5 * standard deviation).

If the error exceeds thresholds is marked as anomalous.

What doesn't work with this approach is that if I have more than one anomalous value in a day, they are not detected. This because error, mean and standard deviation are too much influenced by anomalous values.

How can i fix this problem? Is there another method that i can use to identify thresholds without this issue?

Thank you

  • $\begingroup$ Does your data vary significantly from day to day? $\endgroup$
    – Akavall
    Commented Oct 29, 2019 at 22:24
  • $\begingroup$ no, data are quite similar $\endgroup$
    – Giordano
    Commented Oct 30, 2019 at 8:00
  • $\begingroup$ In the equation: The upper threshold is equals to mean + (5 * standard deviation) are mean and standard deviation of the one day sample or is it of a much larger period? $\endgroup$
    – Akavall
    Commented Oct 30, 2019 at 19:40

1 Answer 1


Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard deviation formula.

You will probably have to change your critical value to something other than 5 to get the same kind of coverage. According to Wikipedia, you'll want the new critical value to be $5\sqrt{\frac{\pi}{2}}$ if your data are iid Gaussian.

An alternative that might be more difficult to implement, but is probably more statistically appropriate, is to use trimmed estimators for the mean and standard deviation. With trimmed estimators, you throw away the most extreme values in your data (the proportion of which is specified beforehand), and estimate your statistics on the remaining data.

The estimator for the mean would be the truncated mean, and the Wikipedia page for trimmed estimators mentions how to get a decent estimator for the standard deviation from the interquartile range.

I hope this helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.