Best anomaly detection algorithm based on two conditions

Question

Choosing the right anomaly detection algorithm seems quite hard at the moment. It might be because I am bombarded with so many alternatives likes clustering, K-Means DBSCAN and so many others.

On my side I have a csv files with thousands of lines, the columns have header that either show the name of the file and the features.

I’m my case this is how the file would look like, keep in mind the file is way larger.

Values columns is the one I want to check about anomalies, in this case I would get a lot of anomalies because the numbers belong to different units-

So first I would need to filter Unit to lets say meters and then check values column data for any anomalies

I would appreciate some advice on what would be the best approach to tackle this problem

Answer 1

Since your data is one-dimensional and numeric, I don't think you need any fancy clustering technique. Clustering is useful when your data points have multiple attributes. When there is just a single attribute, then all you need is a good definition of "anomaly".

For example, suppose you decide that an anomaly is any point more than two standard deviations from the mean. It's easy to find such points using pandas:

import numpy as np
import pandas as pd
from scipy import stats

threshold = 2.0  # in standard deviations
input_file = "path/to/my/file.csv"
target_unit = "meters"

# read the file into a pandas DataFrame
df = pandas.read_csv(input_file)
# filter to only include the target unit
df = df[df['Units']==target_unit]
# compute z-scores for the `values` columns
df['values_z'] = np.absolute(stats.zscore(df['values'].values))
# threshold z-score to identify anomalies
is_anomaly = df['values_z'] > threshold
anomalies = df[is_anomaly]

# `anomalies` is now a DataFrame that contains the anomalous points
# you can consume this however seems appropriate
# for example, you can write the anomalies to a separate file:
anomalies.to_csv('path/to/output.csv')

```