Choosing the right anomaly detection algorithm seems quite hard at the moment. It might be because I am bombarded with so many alternatives likes clustering, K-Means DBSCAN and so many others.

On my side I have a csv files with thousands of lines, the columns have header that either show the name of the file and the features.

I’m my case this is how the file would look like, keep in mind the file is way larger. a

Values columns is the one I want to check about anomalies, in this case I would get a lot of anomalies because the numbers belong to different units-

So first I would need to filter Unit to lets say meters and then check values column data for any anomalies

I would appreciate some advice on what would be the best approach to tackle this problem

  • $\begingroup$ you can start to plot all different value (e.g using scatter plot). It'll give u some idea to define what you consider as an "anomaly". Puting all values in metter is a first great idea, it's called normalisation. $\endgroup$ – akhetos Jan 14 '20 at 15:59
  • $\begingroup$ @akhetos can you give me more information regarding these or any sources that would give me the first push towards this $\endgroup$ – E199504 Jan 14 '20 at 17:48
  • $\begingroup$ @akhetos Visualizing the data is certainly a good idea, but it won't be possible for him to convert all values to meters. Notice that some of the measurements are in celsius $\endgroup$ – zachdj Jan 14 '20 at 20:03
  • $\begingroup$ @zachdj is right. If the only variable you mater is values, this is not possible to compare celsius vs distance. You must separated celsius data and distance data and apply different rules to define anomalie. About distance data, you can look at normalisation to understand why and how do it. $\endgroup$ – akhetos Jan 15 '20 at 10:34
  • $\begingroup$ Also, based on you'r data base, i'm not sure using only "value" has a meaning to define anomalies, since you have data which is 23 millimeter and another data which is 2500 meters.. Could be worth to give additional info about you'r specific problem because with you'r data I have no idea how to make any anomaly detection $\endgroup$ – akhetos Jan 15 '20 at 10:36

Since your data is one-dimensional and numeric, I don't think you need any fancy clustering technique. Clustering is useful when your data points have multiple attributes. When there is just a single attribute, then all you need is a good definition of "anomaly".

For example, suppose you decide that an anomaly is any point more than two standard deviations from the mean. It's easy to find such points using pandas:

import numpy as np
import pandas as pd
from scipy import stats

threshold = 2.0  # in standard deviations
input_file = "path/to/my/file.csv"
target_unit = "meters"

# read the file into a pandas DataFrame
df = pandas.read_csv(input_file)
# filter to only include the target unit
df = df[df['Units']==target_unit]
# compute z-scores for the `values` columns
df['values_z'] = np.absolute(stats.zscore(df['values'].values))
# threshold z-score to identify anomalies
is_anomaly = df['values_z'] > threshold
anomalies = df[is_anomaly]

# `anomalies` is now a DataFrame that contains the anomalous points
# you can consume this however seems appropriate
# for example, you can write the anomalies to a separate file:

  • $\begingroup$ I'm sorry I still didn't have the time to try your code. Can you please add a screenshot of how the result looks like? Will try it out as soon as I get my laptop back $\endgroup$ – E199504 Jan 14 '20 at 20:15
  • $\begingroup$ I can't really generate the results myself since I don't have your csv file :) $\endgroup$ – zachdj Jan 15 '20 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.