How can I find anomalies in each row of data?

Question

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility.

-----------------------------------------------------------------------------
| Facility  | 2017  Jan Visitors | 2017 Feb Visitors | 2017  March Visitors |
-----------------------------------------------------------------------------
| Facility 1|         1234       |       1345        |  100345              |
----------------------------------------------------------------------------
| Facility 2|        56          |      567          | 34                   |
----------------------------------------------------------------------------

How can train this panda dataframe like this row-wise?

Planning to use one class SVM from sklearn. I want to get the anomalies in each facility e.g. in Facility 1 I'd mark 100345 as an anomaly. I have data spanning a couple of years. While we are here I am a super noob in ML and data science can I get a pointer to a condensed tutorial on unsupervised learning most of the ones I am coming across are for supervised.

Answer 1

I am assuming your goal is a time series anomaly detection, in which you want to detect the occurrence of outlier as time goes by. To achieve that, you need to set up a time series prediction model first, and then perform outlier detection base on that.

For unsupervised outlier detection, just check Andrew Ng's machine learning course on Coursera, there is a chapter about anomaly detection based on gaussian kernel. You can also read sklearn docments for OneClassSVM, LocalOutlierFactor, etc. Basically, the core idea of unsupervised anomaly detection is setup a metrics to describe the affinity between samples, and then use a user defined threshold to decide if a new sample is similar enough to other samples.

Answer 2

Assuming no relationships between facilities, the most straightforward way to do outlier detection on your dataframe would be to treat each facility as a separate dataset and look at each one in isolation. So, train one model per facility. In brief, in the unsupervised case you want to fit a model to a dataset that returns a score or measure of distance. You can then use that to judge which points are outliers.

For one class SVM, you could adapt the code and examples on the scikit-learn novelty and outlier detection page or the isolation forest example but bearing in mind that you only need to look at one facility at a time.

If you don't mind not doing unsupervised learning, and willing to consider a simpler approach, plot the distribution of each facility in a histogram and as a timeseries to work out what sorts of numbers of outliers to expect, and at what scale (unless the number of facilities makes this prohibitive).

With an idea of the outliers in the data, points that outside the normal distribution can be identified by transforming each facility's data by z-score and look at the highest and lowest scores; these points are the furthest from the centre of the distribution. For each facility pick a threshold and mark everything outside that threshold as an outlier. This assumes a normal distribution, so it's worth looking at the histograms and line plots to see if that holds in your case.