37

Anomaly Detection or Event Detection can be done in different ways: Basic Way Derivative! If the deviation of your signal from its past & future is high you most probably have an event. This can be extracted by finding large zero crossings in derivative of the signal. Statistical Way Mean of anything is its usual, basic behavior. if something ...


30

Normalization is not always required, but it rarely hurts. Some examples: K-means: K-means clustering is "isotropic" in all directions of space and therefore tends to produce more or less round (rather than elongated) clusters. In this situation leaving variances unequal is equivalent to putting more weight on variables with smaller variance. ...


23

I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks. The package is in Python and its name is pyod. It is published in JMLR. It has multiple algorithms for following individual approaches: Linear Models for Outlier Detection (PCA,vMCD,vOne-Class, and SVM) Proximity-Based Outlier Detection ...


17

h2o has an anomaly detection module and traditionally the code is available in R.However beyond version 3 it has similar module available in python as well,and since h2o is open source it might fit your bill. You can see an working example over here import sys sys.path.insert(1,"../../../") import h2o def anomaly(ip, port): h2o.init(ip, port) ...


16

I recently developed a toolbox: Python Outlier Detection toolbox (PyOD). See GitHub. It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is featured for: Unified APIs, detailed documentation, and interactive examples across various algorithms. Advanced models, including Neural Networks/Deep Learning ...


13

From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural networks that receive as input your data and are trained to output that very same data. The idea is that the training has allowed the net to learn representations of ...


11

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that the error in your test set will be lower due to better generalization. Applying the dropout layer on top of the input layer however throws away a lot of ...


11

The topic you are interest in is called "PU learning" or "positive and unlabeled learning". You can start by having a look into survey literature.


10

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error. When it comes to train the autoencoder on binary data, I agree with you that it can be quite challenging. I suggest to take a look at this blog: https://blog.evjang.com/2016/11/tutorial-categorical-...


8

Standardizing data is recommended because otherwise the range of values in each feature will act as a weight when determining how to cluster data, which is typically undesired. For example consider the standard metric for most clustering algorithms (including DBSCAN in sci-kit learn) -- euclidean, otherwise known as the L2 norm. If one of your features has ...


8

I am currently on same stage like you. I am finding best option for anomaly detection, doing some research. What I have found is I think best matches your need and is better compare to what you have seen. i.e., TwitterAnomalyDetection, SkyLine. I have found better is Numenta's NAB (Numenta Anomaly Benchmark). It also have a very good community support and ...


8

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite the lower visibility) People occasionally argue that there is no difference between an outlier and an anomaly by citing Charu Aggarwal, author of the Book "...


7

Fundamentally there is no difference. Say you have data and you want to build a model of it. As the name suggests, modeling is about finding a model, that is, a simplified representation of your data. In turn, we can view the model as an underlying process that generated your data in the first place, plus some noise. From that point of view, the data you ...


7

There are multiple ways to handle time series abnormalities- If abnormalities are known, build a classification model. Use this model to detect same type of abnormalities for time series data. If abnormalities are unknown, what we have done in our organization is a combination of clustering and classification. First use LOF/K-means/Cook's distance to ...


6

I assume the feature you use to detect abnormality is one row of data in a logfile. If so, Sklearn is your good friend and you can use it as a blackbox. Check the tutorial of one-class SVM and Novelty detection. However, in case that your feature is an entire logfile, you need to first summarize it to some feature of same dimension, and then apply Novealty ...


6

A very robust clustering algorithm against outliers is PFCM from Bezdek. In this paper Bezdek proposes Possibilistic-Fuzzy-C-Means which is an improvement of the different variations of fuzzy posibilistic clustering. This algorithm is particularly good at detecting outliers and avoiding them to influence the clusterization. So using PFCM you could find which ...


6

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output is measured - outliers are expected to have a higher reconstruction error. The main difference seems to be the way how the input is compressed: Plain ...


5

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? Yes, it is always possible to train a classification algorithm when you have labeled i.i.d. training data, and there is no hard reason why you cannot use a simulator to generate that. Whether or not such a trained ...


5

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this on original (assumed to have no outliers) gives you a network which has learnt an abstract representation of the 40 features with 5 features. When outliers show ...


5

You can have a look here, where many open-source algorithms specifically for anomaly detection on time-series data (e.g. metrics) are collected, both for online of offline settings. Almost all of them are unsupervised approaches that require no labels to detect the anomalies. They also automatically handle some of the issues you mentioned, like ...


4

If your Data points are dense and noise points are away from the dense region, you can try DBSCAN algorithm. Tweak its parameters until u get a best fit.


4

Chandola et al's survey is probably the best and most-widely cited survey in the anomaly detection field.


4

Consider how an algorithm might detect a change. You're observing instances of some random variable, $X_1,X_2,\dots,X_{k-1}$. Suddenly (and unknown to you) at $X_k$ something about the distribution of $X$ changes. Now your observations $X_k,\dots,X_n$ are different in some way. You want to know what $k$ is based on your observations alone. In order to ...


4

If you need SQL code that runs various outlier detection methods against any arbitrary table, check out my series of articles and code samples geared towards SQL Server. I provide some preliminary code for Grubb's Tests, Z-Scores and Modified Z-Scores, Interquartile Range, Dixon's Q-Test, GESD, the Tietjen-Moore Test, Pierce's Criterion, Chauvenet's ...


4

you need to distinguish between these cases: Data Imbalance Data Imbalance + Very few number of samples (minority class) Severe Data Imbalance + Very few number of samples (minority class) 20:60 vs. 10:20 vs. 100:1000 vs. 10:100 and these cases: similarities between different classes. wide variations within the same class. You need to understand ...


4

With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. Visually inspect the data for outliers. People are pretty good at this. I would then try to describe a mathematical model to fit the data with the outliers ...


4

You can justify your choices by using data. Treat the anomaly detection like a supervised learning problem where the concept is being anomaly. Then you'll be able to present - for each method - its confusion matrix. Not only it will be a good justification, it will enable to understand the expected results. Many times, we have models and we wonder which ...


4

Without putting in the time to look through Azure's documentation, my guess is that their PCA method is really just a way to do a feature reduction, then use some algorithm they have to classify. Best thing to do is try both methods and then CV and compare performances. https://gallery.azure.ai/Experiment/1219e87f8fb84e88a2e1b54256808bb3


4

I think that heavily depends on the nature of your data (categorical/continuous). I'd start with simple methods first. Those come to my mind: You can compare distribution of each variable either by using quantiles or any statistical test to see whether they are significantly different You could also count occurrence of each label/category and compare them I'...


4

Does that mean that my model (or indeed my approach of using an AE) is ineffective Well, it depends. Auto Encoder are a quite broad field, there are many hyperparameters to tune, width, depth, loss function, optimizer, epochs. How should I proceed? From my gut feeling I would say that you don't have enough data to train the AE properly. Keep in Mind, ...


Only top voted, non community-wiki answers of a minimum length are eligible