37 votes

Open source Anomaly Detection in Python

Anomaly Detection or Event Detection can be done in different ways: Basic Way Derivative! If the deviation of your signal from its past & future is high you most probably have an event. This can ...
user avatar
30 votes
Accepted

Is it necessary to standardize your data before clustering?

Normalization is not always required, but it rarely hurts. Some examples: K-means: K-means clustering is "isotropic" in all directions of space and therefore tends to produce more or less ...
user avatar
23 votes
Accepted

Looking for a good package for anomaly detection in time series

I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks. The package is in Python and its name is pyod. It is published in JMLR. It has ...
user avatar
17 votes

Open source Anomaly Detection in Python

h2o has an anomaly detection module and traditionally the code is available in R.However beyond version 3 it has similar module available in python as well,and since h2o is open source it might fit ...
user avatar
  • 571
17 votes

Open source Anomaly Detection in Python

I recently developed a toolbox: Python Outlier Detection toolbox (PyOD). See GitHub. It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is ...
user avatar
  • 181
13 votes
Accepted

Detecting anomalies with neural network

From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural ...
user avatar
  • 15.4k
11 votes

Validation loss is lower than the training loss

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that ...
user avatar
11 votes
Accepted

Learning with Positive labels only

The topic you are interest in is called "PU learning" or "positive and unlabeled learning". You can start by having a look into survey literature.
user avatar
10 votes
Accepted

Using an autoencoder for anomaly detection on categorical data

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error. When it comes to train the autoencoder ...
user avatar
9 votes

What is the difference between outlier detection and anomaly detection?

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite ...
user avatar
  • 400
8 votes

Is it necessary to standardize your data before clustering?

Standardizing data is recommended because otherwise the range of values in each feature will act as a weight when determining how to cluster data, which is typically undesired. For example consider ...
user avatar
  • 800
8 votes

Open source Anomaly Detection in Python

I am currently on same stage like you. I am finding best option for anomaly detection, doing some research. What I have found is I think best matches your need and is better compare to what you have ...
user avatar
7 votes
Accepted

Difference: Replicator Neural Network vs. Autoencoder

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output ...
user avatar
  • 1,587
7 votes

What is the difference between outlier detection and anomaly detection?

Fundamentally there is no difference. Say you have data and you want to build a model of it. As the name suggests, modeling is about finding a model, that is, a simplified representation of your data....
user avatar
  • 2,118
7 votes

Looking for a good package for anomaly detection in time series

There are multiple ways to handle time series abnormalities- If abnormalities are known, build a classification model. Use this model to detect same type of abnormalities for time series data. If ...
user avatar
6 votes

Open source Anomaly Detection in Python

I assume the feature you use to detect abnormality is one row of data in a logfile. If so, Sklearn is your good friend and you can use it as a blackbox. Check the tutorial of one-class SVM and Novelty ...
user avatar
  • 159
6 votes

What would be a good way to use clustering for outlier detection?

A very robust clustering algorithm against outliers is PFCM from Bezdek. In this paper Bezdek proposes Possibilistic-Fuzzy-C-Means which is an improvement of the different variations of fuzzy ...
user avatar
  • 1,460
5 votes
Accepted

Using simulations to train ML algorithms

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? Yes, it is always possible to train a classification ...
user avatar
  • 27.3k
5 votes
Accepted

Unsupervised feature reduction for anomaly detection with autoencoders

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this ...
user avatar
5 votes

Anomaly detection on time series

You can have a look here, where many open-source algorithms specifically for anomaly detection on time-series data (e.g. metrics) are collected, both for online of offline settings. Almost all of ...
user avatar
  • 370
5 votes
Accepted

Anomaly detection thresholds issue

Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard ...
user avatar
  • 76
4 votes

What would be a good way to use clustering for outlier detection?

If your Data points are dense and noise points are away from the dense region, you can try DBSCAN algorithm. Tweak its parameters until u get a best fit.
user avatar
  • 41
4 votes

What are some good sources to learn fraud/anomaly detection in normal/time-series data?

Chandola et al's survey is probably the best and most-widely cited survey in the anomaly detection field.
user avatar
  • 933
4 votes
Accepted

change detection

Consider how an algorithm might detect a change. You're observing instances of some random variable, $X_1,X_2,\dots,X_{k-1}$. Suddenly (and unknown to you) at $X_k$ something about the distribution of ...
user avatar
4 votes

Tools for automatic anomaly detection on a SQL table?

If you need SQL code that runs various outlier detection methods against any arbitrary table, check out my series of articles and code samples geared towards SQL Server. I provide some preliminary ...
user avatar
4 votes
Accepted

Outlier detection for unbalanced classes

you need to distinguish between these cases: Data Imbalance Data Imbalance + Very few number of samples (minority class) Severe Data Imbalance + Very few number of samples (minority class) 20:60 ...
user avatar
4 votes

Anomaly detection on a time-series data in a CSV format using python

With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. ...
user avatar
  • 166
4 votes
Accepted

Which Outlier Detection Method? Why?

You can justify your choices by using data. Treat the anomaly detection like a supervised learning problem where the concept is being anomaly. Then you'll be able to present - for each method - its ...
user avatar
  • 2,583
4 votes

Which outlier detection can detect these outliers?

You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor. ...
user avatar
4 votes
Accepted

Anomoly detection method selection

Without putting in the time to look through Azure's documentation, my guess is that their PCA method is really just a way to do a feature reduction, then use some algorithm they have to classify. Best ...
user avatar
  • 1,379

Only top scored, non community-wiki answers of a minimum length are eligible