1
$\begingroup$

I have a vector and want to detect outliers in it.

I need an outlier detection method (a non-parametric method) which can detect red points as outliers. enter image description here

Edit: I have a lot of vectors like this. The pattern is similar to this but the values are different. I mean we can see the height since the values are not constant but overall the patterns of outliers against the normal data are like this. This means in one place they beak the continuity of the blues points and go in different directions. Here is another figure. enter image description here Thanks in advance.

Improve this question
$\endgroup$
10
  • $\begingroup$ Take away all the data whose values are close to the 150 mark along the y axis? Can be done using np.where(condition,val1,val2) you can also plot a decision surface accordingly to seperate the data using meshgrids $\endgroup$
    – Aditya
    Mar 3, 2018 at 8:47
  • $\begingroup$ Thanks. But it is not just this vector. I have different vectors but most of them from pattern aspect is like this (not from value aspect). $\endgroup$
    – Arkan
    Mar 3, 2018 at 8:56
  • $\begingroup$ google.co.in/url?sa=t&source=web&rct=j&url=https://… $\endgroup$
    – Aditya
    Mar 3, 2018 at 9:05
  • $\begingroup$ Thanks. I have tested methods like IQR, median and STD but they don't provide desired results. DBScan and methods like that need parameter tunning which will not work for me. $\endgroup$
    – Arkan
    Mar 3, 2018 at 9:22
  • 2
    $\begingroup$ Then use the default parameters and pretend they don't exist. I would try DBScan, or, if your data without outliers are linear, outlier detection through RANSAC. $\endgroup$
    – Elias Strehle
    Mar 3, 2018 at 9:41

1 Answer 1

Reset to default
2
$\begingroup$

Not sure if this is time series data, but it looks like it might be.

For any given narrow window of time there is a distribution that looks like it's centered somewhere a bit higher than y=200 and has a reasonably stable spread. So whatever that distribution is, that's your model.

Then monitor the recent window, and treat that as a sample. Perform a goodness of fit test against the model. If it fails then the points in the window are anomalous.

UPDATE: Based on what you say about the data representing roughly a minute of time, it looks like you could use a window that's about 1 second wide. Run a hypothesis test to see if the data in the window comes from the typical distribution, and if not, just mark all the points in that window as being anomalous. If you need finer resolution than 1s, simply shrink the window.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks. So, how should if define the size of the window? because there is also the possibility of having a window that contains normal and outlier points. So if it fails it will detect all of the points as outliers which means increasing of false positives. $\endgroup$
    – Arkan
    Mar 3, 2018 at 20:55
  • $\begingroup$ Are you concerned about the window's height capturing the typical points, or the width? Are you able to say anything about the dataset and application? That will make it easier to offer relevant ideas. $\endgroup$
    – user10803
    Mar 4, 2018 at 8:18
  • $\begingroup$ Thanks. Width. I have a lot of vectors like this. The pattern is similar to this but the values are different. I mean we can see the height since the values are not constant but overall the patterns of outliers against the normal data are like this. This means in one place they beak the continuity of the blues points and go in different direct $\endgroup$
    – Arkan
    Mar 4, 2018 at 10:41
  • $\begingroup$ Would you mind updating your question with the information you are providing in the comments here? Your remarks here are highly relevant to the approach. Also I would emphasize that if you are able to say anything about the dataset and application, that would be very helpful. (I understand that you may not be able to.) $\endgroup$
    – user10803
    Mar 5, 2018 at 2:26
  • $\begingroup$ I edited. This vector is the output of some processes which can contain some outliers which are caused by some illegal manipulation in data. $\endgroup$
    – Arkan
    Mar 5, 2018 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.