3
$\begingroup$

I have time series data which looks like the graph mentioned below.

I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are that they do not account for the neighboring data points.

For example, in the data show below I do not want to remove the values which are simply maximum or standard deviation away from the mean. I want to remove the points which are circled in the red. The other extreme values are common in the area should not be detected as outlier as these data points have similar nearby data points.

Is there any method to remove these points or is there any python library I can use to remove these points. Normal standard deviation and median filters do not work well for these as they also remove the points which are not circled.

enter image description here

Improve this question
$\endgroup$

5 Answers 5

Reset to default
3
$\begingroup$

You could compute mean and standard deviations in sliding windows, and use those to remove outliers.

For example, taking windows of, say, length 100, you can compute the mean and std for for these 100 successive observations, and see whether any point falls above the 3 sigma rule. In this case, the circled outlier would still be recognized as such, while the others should not, as they are not so outlying w.r.t. neighboring data (i.e. within the window that includes them and their neighboring observations).

Improve this answer
$\endgroup$
4
  • $\begingroup$ That's almost an ARIMA. But yes, pretty much, since models like ARIMA do it, moving average is the common solution for a problem like this. $\endgroup$
    – grochmal
    Commented Jun 7, 2019 at 16:05
  • $\begingroup$ In this case, would it not the points which have similar high values detected as outliers. I mean for the data circled in red will have low values when sliding window approach used wrt threshold than compared to the high values which are not circled in red. Or is it somehitng we have to remove by having not static threshold? $\endgroup$
    – Bhakti
    Commented Jun 11, 2019 at 7:49
  • $\begingroup$ The threshold is computed dynamically on the moving window, so basically your mean and std will differ for every window. The high values at the end wouldn't be detected as the mean will be higher in their respective windows. $\endgroup$
    – rob_med
    Commented Jun 11, 2019 at 11:20
  • $\begingroup$ Is there any method to compute the threshold on moving window? I mean I just knew static threshold and the threshold which was dependent on window statistics but threshold is designed wrt window statistics, would not all points in that window get generalised to that specific statistic? or Did I not understand it correclty? $\endgroup$
    – Bhakti
    Commented Jun 14, 2019 at 12:09
0
$\begingroup$

I don't know of any package that is capable of doing what you want to achieve, but there could be a package.

On way to programmatically deal with this problem is to calculate the difference between two (or multiple for smoothing) consecutive data points. Then you could do filtering based on these values by using a threshold.

Improve this answer
$\endgroup$
0
$\begingroup$

You could first differentiate the series, then apply the classical method based on standard deviation you mentioned.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Differentiate - mean taking the lag values and then applying standard deviation method? $\endgroup$
    – Bhakti
    Commented Jun 11, 2019 at 7:50
  • $\begingroup$ Exactly. It's what the "I" of "ARIMA" means. $\endgroup$
    – Leevo
    Commented Jun 11, 2019 at 7:59
0
$\begingroup$

You could use array slicing. If the data you show here is available in an array you can find the index of the ouliers you want to remove, and simply take them out.

You can also split the data into two different data sets, and apply outlier detection algorithms to the dataset incluing the spike you want removed.

Improve this answer
$\endgroup$
0
$\begingroup$

You can try naive anomaly detection technique (SH-ESD) developed by researchers at Twitter. Here is the link of research paper and implementation of technique in R. There are some python libraries under development.

They have implemented piece-wise median method that is less sensitive to outliers and caught the perfect outliers in the time-series data.

The algorithm is pretty simple to understand.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.