# Tag Info

### Open source Anomaly Detection in Python

Anomaly Detection or Event Detection can be done in different ways: Basic Way Derivative! If the deviation of your signal from its past & future is high you most probably have an event. This can ...
• 6,297
Accepted

### Is it necessary to standardize your data before clustering?

Normalization is not always required, but it rarely hurts. Some examples: K-means: K-means clustering is "isotropic" in all directions of space and therefore tends to produce more or less ...
• 5,275
Accepted

### Looking for a good package for anomaly detection in time series

I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks. The package is in Python and its name is pyod. It is published in JMLR. It has ...

### Open source Anomaly Detection in Python

h2o has an anomaly detection module and traditionally the code is available in R.However beyond version 3 it has similar module available in python as well,and since h2o is open source it might fit ...
• 571

### Open source Anomaly Detection in Python

I recently developed a toolbox: Python Outlier Detection toolbox (PyOD). See GitHub. It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is ...
• 181
Accepted

### Detecting anomalies with neural network

From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural ...
• 15.4k

### Validation loss is lower than the training loss

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that ...
• 9,018
Accepted

### Learning with Positive labels only

The topic you are interest in is called "PU learning" or "positive and unlabeled learning". You can start by having a look into survey literature.
Accepted

### Using an autoencoder for anomaly detection on categorical data

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error. When it comes to train the autoencoder ...
• 901

### What is the difference between outlier detection and anomaly detection?

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite ...
• 400

### Is it necessary to standardize your data before clustering?

Standardizing data is recommended because otherwise the range of values in each feature will act as a weight when determining how to cluster data, which is typically undesired. For example consider ...
• 800

### Open source Anomaly Detection in Python

I am currently on same stage like you. I am finding best option for anomaly detection, doing some research. What I have found is I think best matches your need and is better compare to what you have ...
• 181
Accepted

### Difference: Replicator Neural Network vs. Autoencoder

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output ...
• 1,587

### What is the difference between outlier detection and anomaly detection?

Fundamentally there is no difference. Say you have data and you want to build a model of it. As the name suggests, modeling is about finding a model, that is, a simplified representation of your data....
• 2,118

### Looking for a good package for anomaly detection in time series

There are multiple ways to handle time series abnormalities- If abnormalities are known, build a classification model. Use this model to detect same type of abnormalities for time series data. If ...

### Open source Anomaly Detection in Python

I assume the feature you use to detect abnormality is one row of data in a logfile. If so, Sklearn is your good friend and you can use it as a blackbox. Check the tutorial of one-class SVM and Novelty ...
• 159

### What would be a good way to use clustering for outlier detection?

A very robust clustering algorithm against outliers is PFCM from Bezdek. In this paper Bezdek proposes Possibilistic-Fuzzy-C-Means which is an improvement of the different variations of fuzzy ...
• 1,460
Accepted

### Using simulations to train ML algorithms

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? Yes, it is always possible to train a classification ...
• 27.3k
Accepted

### Unsupervised feature reduction for anomaly detection with autoencoders

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this ...
• 166

### Anomaly detection on time series

You can have a look here, where many open-source algorithms specifically for anomaly detection on time-series data (e.g. metrics) are collected, both for online of offline settings. Almost all of ...
• 370
Accepted

### Anomaly detection thresholds issue

Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard ...
• 76

### What would be a good way to use clustering for outlier detection?

If your Data points are dense and noise points are away from the dense region, you can try DBSCAN algorithm. Tweak its parameters until u get a best fit.
• 41

### What are some good sources to learn fraud/anomaly detection in normal/time-series data?

Chandola et al's survey is probably the best and most-widely cited survey in the anomaly detection field.
• 933
Accepted

### change detection

Consider how an algorithm might detect a change. You're observing instances of some random variable, $X_1,X_2,\dots,X_{k-1}$. Suddenly (and unknown to you) at $X_k$ something about the distribution of ...
• 156

### Tools for automatic anomaly detection on a SQL table?

If you need SQL code that runs various outlier detection methods against any arbitrary table, check out my series of articles and code samples geared towards SQL Server. I provide some preliminary ...
Accepted

### Outlier detection for unbalanced classes

you need to distinguish between these cases: Data Imbalance Data Imbalance + Very few number of samples (minority class) Severe Data Imbalance + Very few number of samples (minority class) 20:60 ...
• 1,839

### Anomaly detection on a time-series data in a CSV format using python

With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. ...
• 166
Accepted

### Which Outlier Detection Method? Why?

You can justify your choices by using data. Treat the anomaly detection like a supervised learning problem where the concept is being anomaly. Then you'll be able to present - for each method - its ...
• 2,583

### Which outlier detection can detect these outliers?

You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor. ...
• 1,103