37

Anomaly Detection or Event Detection can be done in different ways: Basic Way Derivative! If the deviation of your signal from its past & future is high you most probably have an event. This can be extracted by finding large zero crossings in derivative of the signal. Statistical Way Mean of anything is its usual, basic behavior. if something ...


23

Normalization is not always required, but it rarely hurts. Some examples: K-means: K-means clustering is "isotropic" in all directions of space and therefore tends to produce more or less round (rather than elongated) clusters. In this situation leaving variances unequal is equivalent to putting more weight on variables with smaller variance. ...


21

I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks. Package is in python language and name of package is pyod (https://github.com/yzhao062/Pyod). It is published in JMLR. It has multiple algorithms for following individual approaches: Linear Models for Outlier Detection (PCA,vMCD,vOne-Class, ...


17

h2o has an anomaly detection module and traditionally the code is available in R.However beyond version 3 it has similar module available in python as well,and since h2o is open source it might fit your bill. You can see an working example over here import sys sys.path.insert(1,"../../../") import h2o def anomaly(ip, port): h2o.init(ip, port) ...


16

I recently developed a toolbox: Python Outlier Detection toolbox (PyOD). See GitHub. It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is featured for: Unified APIs, detailed documentation, and interactive examples across various algorithms. Advanced models, including Neural Networks/Deep Learning ...


12

From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural networks that receive as input your data and are trained to output that very same data. The idea is that the training has allowed the net to learn representations of ...


11

The topic you are interest in is called "PU learning" or "positive and unlabeled learning". You can start by having a look into survey literature.


10

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error. When it comes to train the autoencoder on binary data, I agree with you that it can be quite challenging. I suggest to take a look at this blog: https://blog.evjang.com/2016/11/tutorial-categorical-...


9

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that the error in your test set will be lower due to better generalization. Applying the dropout layer on top of the input layer however throws away a lot of ...


8

I am currently on same stage like you. I am finding best option for anomaly detection, doing some research. What I have found is I think best matches your need and is better compare to what you have seen. i.e., TwitterAnomalyDetection, SkyLine. I have found better is Numenta's NAB (Numenta Anomaly Benchmark). It also have a very good community support and ...


8

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite the lower visibility) People occasionally argue that there is no difference between an outlier and an anomaly by citing Charu Aggarwal, author of the Book "...


7

Fundamentally there is no difference. Say you have data and you want to build a model of it. As the name suggests, modeling is about finding a model, that is, a simplified representation of your data. In turn, we can view the model as an underlying process that generated your data in the first place, plus some noise. From that point of view, the data you ...


6

I assume the feature you use to detect abnormality is one row of data in a logfile. If so, Sklearn is your good friend and you can use it as a blackbox. Check the tutorial of one-class SVM and Novelty detection. However, in case that your feature is an entire logfile, you need to first summarize it to some feature of same dimension, and then apply Novealty ...


6

Standardizing data is recommended because otherwise the range of values in each feature will act as a weight when determining how to cluster data, which is typically undesired. For example consider the standard metric for most clustering algorithms (including DBSCAN in sci-kit learn) -- euclidean, otherwise known as the L2 norm. If one of your features has ...


6

A very robust clustering algorithm against outliers is PFCM from Bezdek. In this paper Bezdek proposes Possibilistic-Fuzzy-C-Means which is an improvement of the different variations of fuzzy posibilistic clustering. This algorithm is particularly good at detecting outliers and avoiding them to influence the clusterization. So using PFCM you could find which ...


6

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output is measured - outliers are expected to have a higher reconstruction error. The main difference seems to be the way how the input is compressed: Plain ...


6

There are multiple way to handle time series abnormalities- 1) If abnormalities are known, build a classification model. Use this model to detect same type of abnormalities for time series data. 2) If abnormalities are unknown, What we have done in our organisation- is a combination of clustering and classification- First use LOF/K-means/Cook's ...


5

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? Yes, it is always possible to train a classification algorithm when you have labeled i.i.d. training data, and there is no hard reason why you cannot use a simulator to generate that. Whether or not such a trained ...


5

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this on original (assumed to have no outliers) gives you a network which has learnt an abstract representation of the 40 features with 5 features. When outliers show ...


5

You can have a look here, where many open-source algorithms specifically for anomaly detection on time-series data (e.g. metrics) are collected, both for online of offline settings. Almost all of them are unsupervised approaches that require no labels to detect the anomalies. They also automatically handle some of the issues you mentioned, like ...


4

If your Data points are dense and noise points are away from the dense region, you can try DBSCAN algorithm. Tweak its parameters until u get a best fit.


4

you need to distinguish between these cases: Data Imbalance Data Imbalance + Very few number of samples (minority class) Severe Data Imbalance + Very few number of samples (minority class) 20:60 vs. 10:20 vs. 100:1000 vs. 10:100 and these cases: similarities between different classes. wide variations within the same class. You need to understand ...


4

Consider how an algorithm might detect a change. You're observing instances of some random variable, $X_1,X_2,\dots,X_{k-1}$. Suddenly (and unknown to you) at $X_k$ something about the distribution of $X$ changes. Now your observations $X_k,\dots,X_n$ are different in some way. You want to know what $k$ is based on your observations alone. In order to ...


4

With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. Visually inspect the data for outliers. People are pretty good at this. I would then try to describe a mathematical model to fit the data with the outliers ...


4

Without putting in the time to look through Azure's documentation, my guess is that their PCA method is really just a way to do a feature reduction, then use some algorithm they have to classify. Best thing to do is try both methods and then CV and compare performances. https://gallery.azure.ai/Experiment/1219e87f8fb84e88a2e1b54256808bb3


4

I think that heavily depends on the nature of your data (categorical/continuous). I'd start with simple methods first. Those come to my mind: You can compare distribution of each variable either by using quantiles or any statistical test to see whether they are significantly different You could also count occurrence of each label/category and compare them I'...


4

This question is quite broad. I'll try to set you on the right path, more so than providing a truly complete answer. Theoretical background As others have mentioned, the task you're trying to do is usually known as anomaly detection, also known as novelty detection. There's many possible ways to approach this kind of task, depending on the assumptions you ...


4

I recently had a similar problem (removing abnormal peaks from a time series). That's what I suggest you: Get the smoothed trend. There are several techniques you can employ, such as various forms of exponential smoothing. Find the difference between actual trend observations and smoothed ones. Normalize this distribution of distances (using Z-score, i.e. ...


4

Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard deviation formula. You will probably have to change your critical value to something other than 5 to get the same kind of coverage. According to Wikipedia, you'll ...


4

Since your data is one-dimensional and numeric, I don't think you need any fancy clustering technique. Clustering is useful when your data points have multiple attributes. When there is just a single attribute, then all you need is a good definition of "anomaly". For example, suppose you decide that an anomaly is any point more than two standard ...


Only top voted, non community-wiki answers of a minimum length are eligible