# Unsupervised learning for anomaly detection

I've started working on an anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor making machines.

My dataset looks like this:

ContextID   Time_ms Ar_Flow_sccm    BacksGas_Flow_sccm
7289973 09:12:48.502    49.56054688 1.953125
7289973 09:12:48.603    49.56054688 2.05078125
7289973 09:12:48.934    99.85351563 2.05078125
7289973 09:12:49.924    351.3183594 2.05078125
7289973 09:12:50.924    382.8125    1.953125
7289973 09:12:51.924    382.8125    1.7578125
7289973 09:12:52.934    382.8125    1.7578125
7289999 09:15:36.434    50.04882813 1.7578125
7289999 09:15:36.654    50.04882813 1.7578125
7289999 09:15:36.820    50.04882813 1.66015625
7289999 09:15:37.904    333.2519531 1.85546875
7289999 09:15:38.924    377.1972656 1.953125
7289999 09:15:39.994    377.1972656 1.7578125
7289999 09:15:41.94     388.671875  1.85546875
7289999 09:15:42.136    388.671875  1.85546875
7290025 09:18:00.429    381.5917969 1.85546875
7290025 09:18:01.448    381.5917969 1.85546875
7290025 09:18:02.488    381.5917969 1.953125
7290025 09:18:03.549    381.5917969 14.453125
7290025 09:18:04.589    381.5917969 46.77734375


What I have to do is to apply some unsupervised learning technique on each and every parameter column individually and find any anomalies that might exist in there. The ContextID is more like a product number.

I would like to know which unsupervised learning techniques can be used for this kind of task at hand.

Thanks

• Do you know the percent of anomalies in your data? Mar 21 '19 at 11:33
• I am afraid not @AlirezaZolanvari Mar 21 '19 at 11:43

You can use a density based method such as Local Outlier Factor in order to do this.

If you want to identify outliers in each column independent of other columns, then you would apply this method separately to each column. (Note that this is different from identifying outlier data points considering all columns together.)

You can use Cluster analysis-based outlier detection or autoencoders for instance. Please look at this post: Anomaly detection on time series for a similar answer about time series.