1
$\begingroup$

As is clear from the figure, the blue points, which don't follow the trend, are anomalous points.

I'm wondering about the best non-parametric method to detect those points. I have tested some outlier detection methods such as standard deviation, etc. but they don't provide good results while it is clear from the figure.

enter image description here

Improve this question
$\endgroup$

4 Answers 4

Reset to default
1
$\begingroup$

In your example what differentiates the clusters is not the raw value but rapid departure from previous points. I might look into change-point detection. Nonparametic, but still requires some fiddling with tuning parameters.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks. Can you name some functions about change point detection so I can test them. They give just change point or define upper and lower level of change? $\endgroup$
    – Arkan
    Dec 14, 2017 at 3:45
1
$\begingroup$

I would recommend a rolling average, it can be quite robust and is not upset by slow changes over time. You then can use your existing data to determine at which level of deviation you want an alarm. This optimization depends on weither you want rather to catch most of the deviations or minimize the number of false positive alarms.

EDIT1: In the end you will always need to tweak parameters. There is no real ground truth. An anomaly is always a subjective thing.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you @El Burro. Can you explain a bit more, so I can implement it and test it in matlab or python. $\endgroup$
    – Arkan
    Dec 14, 2017 at 15:30
  • $\begingroup$ On what part do you need more information? $\endgroup$
    – El Burro
    Dec 15, 2017 at 10:01
  • $\begingroup$ Rolling average methods to be used on this case. $\endgroup$
    – Arkan
    Dec 15, 2017 at 17:06
0
$\begingroup$

A solution can be using DBSCAN algorithm to cluster data. Then, if you set a proper radius for the DBSCAN algorithm, you will get three clusters. Therefore, you can detect some clusters as anomaly that number of their members is less than a threshold.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks. But it is not possible to set a threshold since the number of anomalous points is not constant and definite. $\endgroup$
    – Arkan
    Dec 14, 2017 at 2:47
  • $\begingroup$ @Arkan ok. but, you should have a threshold to find the anomalous points. Moreover, you can set a threshold and number of members is dynamic too. There is no conflict between these two. $\endgroup$
    – OmG
    Dec 14, 2017 at 2:51
  • $\begingroup$ Thank you for your reply. But as mentioned in the post I'm looking for a non-parametric method. Isn't there any other method except DBSCAN which is like standard deviation method to be able to detect such points? BTW, DBSCAN needs two important parameter minpts and epsilon which makes it a parametric method in action and need tuning $\endgroup$
    – Arkan
    Dec 14, 2017 at 3:14
0
$\begingroup$

Try modeling the time series with a midpoint estimate (expected value at each time) and a band estimate (permissible spread at any given midpoint). Then being an anomaly is just a matter of being outside the band.

How you estimate the midpoints depends on the problem. For example if your time series has one or more seasonalities, you could use time series decomposition (e.g., seasonal-trend decomposition using loess). The band estimate is also problem-dependent.

I did some work like this for a bookings data time series. I wrote about it here: https://techblog.expedia.com/2016/07/28/applying-data-science-to-monitoring/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.