3
$\begingroup$

I've started working on an anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor making machines.

My dataset looks like this:

ContextID   Time_ms Ar_Flow_sccm    BacksGas_Flow_sccm
7289973 09:12:48.502    49.56054688 1.953125
7289973 09:12:48.603    49.56054688 2.05078125
7289973 09:12:48.934    99.85351563 2.05078125
7289973 09:12:49.924    351.3183594 2.05078125
7289973 09:12:50.924    382.8125    1.953125
7289973 09:12:51.924    382.8125    1.7578125
7289973 09:12:52.934    382.8125    1.7578125
7289999 09:15:36.434    50.04882813 1.7578125
7289999 09:15:36.654    50.04882813 1.7578125
7289999 09:15:36.820    50.04882813 1.66015625
7289999 09:15:37.904    333.2519531 1.85546875
7289999 09:15:38.924    377.1972656 1.953125
7289999 09:15:39.994    377.1972656 1.7578125
7289999 09:15:41.94     388.671875  1.85546875
7289999 09:15:42.136    388.671875  1.85546875
7290025 09:18:00.429    381.5917969 1.85546875
7290025 09:18:01.448    381.5917969 1.85546875
7290025 09:18:02.488    381.5917969 1.953125
7290025 09:18:03.549    381.5917969 14.453125
7290025 09:18:04.589    381.5917969 46.77734375

What I have to do is to apply some unsupervised learning technique on each and every parameter column individually and find any anomalies that might exist in there. The ContextID is more like a product number.

I would like to know which unsupervised learning techniques can be used for this kind of task at hand.

Thanks

Improve this question
$\endgroup$
2
  • $\begingroup$ Do you know the percent of anomalies in your data? $\endgroup$
    – Alireza Zolanvari
    Mar 21, 2019 at 11:33
  • $\begingroup$ I am afraid not @AlirezaZolanvari $\endgroup$
    – some_programmer
    Mar 21, 2019 at 11:43

3 Answers 3

Reset to default
1
$\begingroup$

You can use a density based method such as Local Outlier Factor in order to do this.

If you want to identify outliers in each column independent of other columns, then you would apply this method separately to each column. (Note that this is different from identifying outlier data points considering all columns together.)

Improve this answer
$\endgroup$
0
$\begingroup$

You can use Cluster analysis-based outlier detection or autoencoders for instance. Please look at this post: Anomaly detection on time series for a similar answer about time series.

Improve this answer
$\endgroup$
0
$\begingroup$

You can consider this package, pyod. It has various anomaly/outlier detection algorithms all in one package.

Approaches :

  1. Use a dimension reduction technique like tsne, which can collapse high D into 2 or 3 D space and plot it out. You will visually be able to find clusters of points which are away from every other points, signifying an outlier. In your case, it is just 3D data, you can directly plot them in a 3D plot.
  2. After which run pyod by signifying the outlier ratio, and ideally it should find it. It has various algos like isolation forest, COPOD etc.

Hope it helps.
Cheers

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.