25
votes
Accepted
Looking for a good package for anomaly detection in time series
I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks.
The package is in Python and its name is pyod. It is published in JMLR.
It has ...
- 366
18
votes
Open source Anomaly Detection in Python
I recently developed a toolbox: Python Outlier Detection toolbox (PyOD). See GitHub.
It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is ...
- 191
13
votes
Accepted
Detecting anomalies with neural network
From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural ...
- 23.8k
11
votes
Validation loss is lower than the training loss
It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that ...
- 9,348
11
votes
Accepted
Learning with Positive labels only
The topic you are interest in is called "PU learning" or "positive and unlabeled learning".
You can start by having a look into survey literature.
- 1,014
10
votes
Accepted
Using an autoencoder for anomaly detection on categorical data
In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error.
When it comes to train the autoencoder ...
- 921
9
votes
What is the difference between outlier detection and anomaly detection?
(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite ...
- 400
8
votes
Open source Anomaly Detection in Python
I am currently on same stage like you. I am finding best option for anomaly detection, doing some research.
What I have found is I think best matches your need and is better compare to what you have ...
- 181
8
votes
Accepted
Difference: Replicator Neural Network vs. Autoencoder
Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output ...
- 1,637
7
votes
What is the difference between outlier detection and anomaly detection?
Fundamentally there is no difference. Say you have data and you want to build a model of it. As the name suggests, modeling is about finding a model, that is, a simplified representation of your data....
- 2,238
7
votes
Looking for a good package for anomaly detection in time series
There are multiple ways to handle time series abnormalities-
If abnormalities are known, build a classification model. Use this model to detect same type of abnormalities for time series data.
If ...
- 415
6
votes
Accepted
Using simulations to train ML algorithms
Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online?
Yes, it is always possible to train a classification ...
- 28.7k
5
votes
Accepted
Unsupervised feature reduction for anomaly detection with autoencoders
I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this ...
- 166
5
votes
Anomaly detection on time series
You can have a look here, where many open-source algorithms specifically for anomaly detection on time-series data (e.g. metrics) are collected, both for online of offline settings.
Almost all of ...
- 480
5
votes
Accepted
Anomaly detection thresholds issue
Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard ...
- 76
4
votes
Anomaly detection on a time-series data in a CSV format using python
With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. ...
- 166
4
votes
Accepted
Which Outlier Detection Method? Why?
You can justify your choices by using data.
Treat the anomaly detection like a supervised learning problem where the concept is being anomaly.
Then you'll be able to present - for each method - its ...
- 2,623
4
votes
Accepted
Outlier detection for unbalanced classes
you need to distinguish between these cases:
Data Imbalance
Data Imbalance + Very few number of samples (minority class)
Severe Data Imbalance + Very few number of samples (minority class)
20:60 ...
- 1,969
4
votes
Accepted
change detection
Consider how an algorithm might detect a change. You're observing instances of some random variable, $X_1,X_2,\dots,X_{k-1}$. Suddenly (and unknown to you) at $X_k$ something about the distribution of ...
- 156
4
votes
Tools for automatic anomaly detection on a SQL table?
If you need SQL code that runs various outlier detection methods against any arbitrary table, check out my series of articles and code samples geared towards SQL Server. I provide some preliminary ...
- 371
4
votes
What are some good sources to learn fraud/anomaly detection in normal/time-series data?
Chandola et al's survey is probably the best and most-widely cited survey in the anomaly detection field.
- 933
4
votes
Which outlier detection can detect these outliers?
You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor.
...
- 1,113
4
votes
Accepted
Anomoly detection method selection
Without putting in the time to look through Azure's documentation, my guess is that their PCA method is really just a way to do a feature reduction, then use some algorithm they have to classify. Best ...
- 1,429
4
votes
Detecting anomalies with neural network
I think that heavily depends on the nature of your data (categorical/continuous). I'd start with simple methods first. Those come to my mind:
You can compare distribution of each variable either by ...
- 1,669
4
votes
Accepted
Autoencoder for anomaly detection from feature vectors
Does that mean that my model (or indeed my approach of using an AE) is ineffective
Well, it depends. Auto Encoder are a quite broad field, there are many hyperparameters to tune, width, depth, loss ...
- 176
4
votes
To detect unauthorized access using outlier detection
This question is quite broad. I'll try to set you on the right path, more so than providing a truly complete answer.
Theoretical background
As others have mentioned, the task you're trying to do is ...
- 279
4
votes
Multi Class + Negative Class Image Classification Strategies
There is no general strategy to anything in machine learning. Moreover, research is scattered over multiple domains so it becomes harder to get your head around, "the best strategy for a case". You ...
- 820
4
votes
Why use Variational Autoencoders VAE instead of Autoencoders AE in Anomaly Detection?
Variational autoencoders encourage the model to generalize features and reconstruct images as an aggregation of those features. This is what the latent space encodes, a compressed feature vector.
...
4
votes
Methods to detect this kind of outliers
I recently had a similar problem (removing abnormal peaks from a time series). That's what I suggest you:
Get the smoothed trend. There are several techniques you can employ, such as various forms of ...
- 6,095
4
votes
Best anomaly detection algorithm based on two conditions
Since your data is one-dimensional and numeric, I don't think you need any fancy clustering technique. Clustering is useful when your data points have multiple attributes. When there is just a ...
- 2,654
Only top scored, non community-wiki answers of a minimum length are eligible