Skip to main content
Commonmark migration
Source Link

Background
I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why I'm posting this here, don't even know if this is appropriate or not.

Problem I have some 2D data in pairs of (x,y). I only have the data and don't know anything about what kind of functional feature the data have. Now this data randomly have some wrong values or (should I say) outliers. And I have to fix them before working with that data. Lets show an example: [![enter image description here][1]][1]enter image description here

As you can see the data have a nice uniform feature except for the 3rd point, and obviously that's a wrong point. I have fix this kind of points. You can see If I can just detect that point then I can just delete and fit it with surrounding (e.g with a spline). Now I have lots of data just like this, and plotting every time to check and then manually detect and smooth those points is really tiresome.

Now I don't have any idea about how to proceed with this problem. I tried to search but was lost in the huge world of machine learning. So, what kind of methods and technique do I have to learn, so that I can aromatically detect those points. You can show me a sample code (in python preferably) to do this kind of jobs. Thanks. [1]: https://i.sstatic.net/kGXn3.png

Background
I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why I'm posting this here, don't even know if this is appropriate or not.

Problem I have some 2D data in pairs of (x,y). I only have the data and don't know anything about what kind of functional feature the data have. Now this data randomly have some wrong values or (should I say) outliers. And I have to fix them before working with that data. Lets show an example: [![enter image description here][1]][1]

As you can see the data have a nice uniform feature except for the 3rd point, and obviously that's a wrong point. I have fix this kind of points. You can see If I can just detect that point then I can just delete and fit it with surrounding (e.g with a spline). Now I have lots of data just like this, and plotting every time to check and then manually detect and smooth those points is really tiresome.

Now I don't have any idea about how to proceed with this problem. I tried to search but was lost in the huge world of machine learning. So, what kind of methods and technique do I have to learn, so that I can aromatically detect those points. You can show me a sample code (in python preferably) to do this kind of jobs. Thanks. [1]: https://i.sstatic.net/kGXn3.png

Background
I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why I'm posting this here, don't even know if this is appropriate or not.

Problem I have some 2D data in pairs of (x,y). I only have the data and don't know anything about what kind of functional feature the data have. Now this data randomly have some wrong values or (should I say) outliers. And I have to fix them before working with that data. Lets show an example: enter image description here

As you can see the data have a nice uniform feature except for the 3rd point, and obviously that's a wrong point. I have fix this kind of points. You can see If I can just detect that point then I can just delete and fit it with surrounding (e.g with a spline). Now I have lots of data just like this, and plotting every time to check and then manually detect and smooth those points is really tiresome.

Now I don't have any idea about how to proceed with this problem. I tried to search but was lost in the huge world of machine learning. So, what kind of methods and technique do I have to learn, so that I can aromatically detect those points. You can show me a sample code (in python preferably) to do this kind of jobs. Thanks.

Notice removed Authoritative reference needed by CommunityBot
Bounty Ended with Mario's answer chosen by CommunityBot
Notice added Authoritative reference needed by Eular
Bounty Started worth 50 reputation by Eular
Source Link
Eular
  • 71
  • 6

Methods to detect this kind of outliers

Background
I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why I'm posting this here, don't even know if this is appropriate or not.

Problem I have some 2D data in pairs of (x,y). I only have the data and don't know anything about what kind of functional feature the data have. Now this data randomly have some wrong values or (should I say) outliers. And I have to fix them before working with that data. Lets show an example: [![enter image description here][1]][1]

As you can see the data have a nice uniform feature except for the 3rd point, and obviously that's a wrong point. I have fix this kind of points. You can see If I can just detect that point then I can just delete and fit it with surrounding (e.g with a spline). Now I have lots of data just like this, and plotting every time to check and then manually detect and smooth those points is really tiresome.

Now I don't have any idea about how to proceed with this problem. I tried to search but was lost in the huge world of machine learning. So, what kind of methods and technique do I have to learn, so that I can aromatically detect those points. You can show me a sample code (in python preferably) to do this kind of jobs. Thanks. [1]: https://i.sstatic.net/kGXn3.png