Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
72 votes
7 answers
82k views

Open source Anomaly Detection in Python

Problem Background: I am working on a project that involves log files similar to those found in the IT monitoring space (to my best understanding of IT space). These log files are time-series data, ...
user avatar
  • 933
34 votes
5 answers
55k views

Is it necessary to standardize your data before clustering?

Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line: ...
user avatar
  • 791
22 votes
4 answers
22k views

Looking for a good package for anomaly detection in time series

Is there a comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series? There is a one class SVM package in scikit-learn but it is not for the ...
user avatar
  • 1,207
14 votes
4 answers
14k views

Detecting anomalies with neural network

I have a large multi dimensional dataset that is generated each day. What would be a good approach to detect any kind of 'anomaly' as compared with previous days? Is this a suitable problem that could ...
user avatar
  • 641
13 votes
4 answers
8k views

What is the difference between outlier detection and anomaly detection?

I would like to know the difference in terms of applications (e.g. which one is credit card fraud detection?) and in terms of used techniques. Example papers which define the task would be welcome.
user avatar
  • 17.5k
11 votes
1 answer
776 views

Learning with Positive labels only

I have ~7 million rows of customer data (~500 sparse attributes) A million out of them have opted in to a new service. How do I use this signal to predict which of the remaining customers are likely ...
user avatar
11 votes
2 answers
6k views

Tools for automatic anomaly detection on a SQL table?

I have a large SQL table that is essentially a log. The data is pretty complex and I'm trying to find some way to identify anomalies without me understanding all the data. I've found lots of tools for ...
user avatar
9 votes
1 answer
11k views

Validation loss is lower than the training loss

I am using autoencoder for anomaly detection in warranty data. Architecture 1: The plot shows the training vs validation loss based on Architecture 1. As we see in the plot, validation loss is ...
user avatar
  • 235
9 votes
1 answer
7k views

Difference: Replicator Neural Network vs. Autoencoder

I'm currently studying papers about outlier detection using RNN's (Replicator Neural Networks) and wonder what is the particular difference to Autoencoders? RNN's seem to be treaded for many as the ...
user avatar
  • 275
8 votes
1 answer
7k views

Using an autoencoder for anomaly detection on categorical data

Say a dataset has 0.5% of its features continuous and 99.5% categorical (binary) with ~2400 features in total. In this dataset, each observation is 1 of 2 classes - Fraud (1) or Not Fraud (0). ...
user avatar
  • 1,494
8 votes
3 answers
6k views

Isolation forest sklearn contamination param

I am working on an unsupervised anomaly detection task on time series data using an isolation forest algorithm. I am developing it in Python, more in detail using ...
user avatar
  • 325
8 votes
1 answer
5k views

how to compare different sets of time series data

I am trying to do some anomaly detection between time#series using Python and sklearn (but other package suggestions are definitely welcome!). I have a set of 10 time-series; each time-series ...
user avatar
7 votes
5 answers
2k views

What would be a good way to use clustering for outlier detection?

For simplicity let's assume the feature space is the XY plane.
user avatar
7 votes
3 answers
23k views

Anomaly detection on time series

I've just started working on an anomaly detection development in Python. My data sets are a collection of timeseries. More in details, data are coming from some sensors/meters which record and ...
user avatar
  • 325
7 votes
3 answers
1k views

Which outlier detection can detect these outliers?

I have a vector and want to detect outliers in it. The following figure shows the distribution of the vector. Red points are outliers. Blue points are normal points. Yellow points are also normal. ...
user avatar
  • 443
6 votes
2 answers
4k views

What methods can be used to detect anomalies in temporal texual data?

I've been looking for methods that can help figure out anomalies in textual data stored in databases. Major goal is to use a unsupervised learning method to detect the anomalies. Further how can I ...
user avatar
6 votes
1 answer
382 views

How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of ...
user avatar
6 votes
1 answer
2k views

Netflow anomaly detection python packages

Is anyone aware of any open source / python packages for Netflow Anomaly detection ? I found some on github but anyone who has more experience with it. please advise.
user avatar
  • 61
5 votes
3 answers
2k views

To detect unauthorized access using outlier detection

I am working on project where my task is to find unauthorized access using any machine learning technique. Let me clear my problem definition. UserA access website using chrome browser from windows ...
user avatar
5 votes
3 answers
4k views

change detection

I have a question related to change detection. Application domain is robotics/planning. Background/setting: There is a sensor detecting distance from obstacle (ultrasonic / sonar sensor) at a ...
user avatar
  • 153
5 votes
1 answer
637 views

Unsupervised feature reduction for anomaly detection with autoencoders

I am collecting a big number of generated numeric features for the task of unsupervised anomaly detection. I can assume that all training data is considered normal. I expect some of the generated ...
user avatar
  • 153
5 votes
1 answer
2k views

Using simulations to train ML algorithms

Possibly similar question: Is it ok to collect data using algorithm to train another? I have a model that accurately describes an underlying physical, complex, system. The model is basically a set of ...
user avatar
5 votes
2 answers
9k views

How would I apply anomaly detection to time series data in LSTM?

I am using a LSTM RNN in Python and have successfully completed the prediction phase. My ultimate goal is anomaly detection. I'm hoping to have something like what you could see on Facebook Prophet, ...
user avatar
  • 51
5 votes
1 answer
1k views

Multi Class + Negative Class Image Classification Strategies

I have seen a recurring theme in real-world problems I've worked with, where the problem looks something like "build an image classifier that recognizes classes A, B, and C but if the input is not ...
user avatar
  • 151
5 votes
2 answers
3k views

Unsupervised Anomaly Detection in Images

I would like to detect defects/anomalies in images. Due to the lack of images with anomalies, I try to solve the problem in an unsupervised manner. Until now, I trained a variational autoencoder ...
user avatar
  • 51
5 votes
1 answer
4k views

Anomaly detection for transaction data

I have transaction details for credit data (bank transfers, peer to peer transfers, etc). Currently, I have one year worth of data which I cannot properly classify. I'm looking for input and ...
user avatar
  • 51
5 votes
1 answer
3k views

Isolation Forest Feature Importance

As of scikit-learn version 0.19.1, there is no implementation for calculating feature importance in an Isolation Forest. I'm also having trouble finding any online resources proposing ways to get at ...
user avatar
5 votes
2 answers
81 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
user avatar
5 votes
1 answer
695 views

Time Series pattern recognition and classification problem

I have some labeled sensor data. Now, I would like to know how to extract features from time series using DFT, DWT, and HAAR transforms. I know that the transformations above transform a signal to ...
user avatar
5 votes
2 answers
285 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
user avatar
4 votes
1 answer
5k views

Multivariate outlier detection with isolation forest..How to detect most effective features?

I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in http://scikit-learn.org/stable/auto_examples/ensemble/...
user avatar
4 votes
1 answer
4k views

How to find patterns in a series of timestamps

I have a series of timestamps that represent the time a user clicked a certain button. My goal is to detect the automated clicks, so I need to find recurring patterns in the data that may point to an ...
user avatar
  • 121
4 votes
2 answers
858 views

Anomaly detection in multiple parameters

I am a newbie to data science with a typical problem. I have a data set with metric1, metric2 and metric3. All these metrics are interdependent on each other. I want to detect anomalies in metric3. ...
user avatar
4 votes
1 answer
1k views

Interpretation of scikit-learn one class svm scores

How can I interpret the scores generated by the function score_samples(X) from a scikit-learn OneClassSVM model? Is there a way ...
user avatar
  • 141
4 votes
1 answer
73 views

How to construct confidence bound for Time Series Prediction?

I have some time series data and am using some Deep Learning techniques to get its prediction. Now, I would like to construct confidence bounds for it. I calculated the residuals, their mean and ...
user avatar
  • 141
4 votes
1 answer
71 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
user avatar
  • 585
4 votes
1 answer
509 views

If a time series has random time events, how to detect patterns?

My app receives messages with a random number of bits at a random time. But two weeks ago I started to notice some almost regular patterns on the metrics of my app. I suspect they are some bots ...
user avatar
3 votes
2 answers
954 views

Which Outlier Detection Method? Why?

For detecting an outlier in a vector I have tested different well known outlier detection methods. Finally, I used combination of different methods and an agreement between those methods. Now, a ...
user avatar
  • 443
3 votes
2 answers
1k views

What are some good sources to learn fraud/anomaly detection in normal/time-series data?

I would like to know more on fraud/anomaly detection. I am looking for good source or survey article/book etc out there which will give me some preliminary idea of the area. Any suggestion is ...
user avatar
  • 1,057
3 votes
2 answers
3k views

Outlier detection for unbalanced classes

I have to make a predictive model for predicting a boolean Won/Lost variable based on some other numeric data; and further find out the features of observations that have 'Won'. However, the number ...
user avatar
  • 413
3 votes
1 answer
314 views

Best anomaly detection algorithm based on two conditions

Choosing the right anomaly detection algorithm seems quite hard at the moment. It might be because I am bombarded with so many alternatives likes clustering, K-Means DBSCAN and so many others. On my ...
user avatar
  • 585
3 votes
4 answers
691 views

Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...
user avatar
  • 585
3 votes
2 answers
6k views

Autoencoder for anomaly detection from feature vectors

I am trying to use an autoencoder (as described here https://blog.keras.io/building-autoencoders-in-keras.html#) for anomaly detection. I am using a ~1700 feature vector (rather than images, which ...
user avatar
  • 584
3 votes
4 answers
177 views

Definition of "inside" in K-means?

After conducting a cluster analysis using K-means, I have new data coming online that I need to detect anomalies with. Anomalies are assumed to not be within the clusters. So, how is one to define "...
user avatar
  • 791
3 votes
1 answer
80 views

Outlier/Anomaly Detection History

I have been reading about different methods of anomaly detection, their structure and the way they work. Recently I have been trying to find some scholar articles, writings or books where I can learn ...
user avatar
  • 585
3 votes
5 answers
5k views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
user avatar
  • 151
3 votes
1 answer
67 views

How to determine the abnormality of a specific variable by taking into account all the other variables in the data?

I have an issue of machine learning/anomaly detection. Indeed, I have a variable Y and several other variables X. The purpose is to quantify the degree of abnormality of the data on Y but I have to ...
user avatar
  • 31
3 votes
1 answer
271 views

How to classify parametric curves?

I am working on a project which aims at determining whether a patient has cervical issues or not, based on a certain movement (for instance, turning the head from left to right and so on). For each ...
user avatar
  • 133
3 votes
2 answers
515 views

Unsupervised learning for anomaly detection

I've started working on an anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor making machines. ...
user avatar
  • 187
3 votes
2 answers
2k views

Outlier detection with sklearn

I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the ...
user avatar
  • 5,654

1
2 3 4 5
7