Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
25 votes
Accepted

Looking for a good package for anomaly detection in time series

I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks. The package is in Python and its name is pyod. It is published in JMLR. It has ...
Shankar Chavan's user avatar
18 votes

Open source Anomaly Detection in Python

I recently developed a toolbox: Python Outlier Detection toolbox (PyOD). See GitHub. It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is ...
Yue Zhao's user avatar
  • 191
13 votes
Accepted

Detecting anomalies with neural network

From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural ...
noe's user avatar
  • 26.9k
11 votes

Validation loss is lower than the training loss

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that ...
Jan van der Vegt's user avatar
11 votes
Accepted

Learning with Positive labels only

The topic you are interest in is called "PU learning" or "positive and unlabeled learning". You can start by having a look into survey literature.
Graph4Me Consultant's user avatar
10 votes
Accepted

Using an autoencoder for anomaly detection on categorical data

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error. When it comes to train the autoencoder ...
Andreas Look's user avatar
9 votes

What is the difference between outlier detection and anomaly detection?

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite ...
Marco13's user avatar
  • 400
8 votes
Accepted

Difference: Replicator Neural Network vs. Autoencoder

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output ...
stmax's user avatar
  • 1,637
7 votes

What is the difference between outlier detection and anomaly detection?

Fundamentally there is no difference. Say you have data and you want to build a model of it. As the name suggests, modeling is about finding a model, that is, a simplified representation of your data....
tom's user avatar
  • 2,248
7 votes

Looking for a good package for anomaly detection in time series

There are multiple ways to handle time series abnormalities- If abnormalities are known, build a classification model. Use this model to detect same type of abnormalities for time series data. If ...
Arpit Sisodia's user avatar
6 votes
Accepted

Using simulations to train ML algorithms

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online? Yes, it is always possible to train a classification ...
Neil Slater's user avatar
5 votes
Accepted

Unsupervised feature reduction for anomaly detection with autoencoders

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this ...
rishiehari's user avatar
5 votes

Anomaly detection on time series

You can have a look here, where many open-source algorithms specifically for anomaly detection on time-series data (e.g. metrics) are collected, both for online of offline settings. Almost all of ...
rob_med's user avatar
  • 480
5 votes
Accepted

Anomaly detection thresholds issue

Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard ...
goopy's user avatar
  • 76
4 votes

Anomaly detection on a time-series data in a CSV format using python

With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. ...
Eric S's user avatar
  • 166
4 votes
Accepted

Which Outlier Detection Method? Why?

You can justify your choices by using data. Treat the anomaly detection like a supervised learning problem where the concept is being anomaly. Then you'll be able to present - for each method - its ...
DaL's user avatar
  • 2,633
4 votes
Accepted

Outlier detection for unbalanced classes

you need to distinguish between these cases: Data Imbalance Data Imbalance + Very few number of samples (minority class) Severe Data Imbalance + Very few number of samples (minority class) 20:60 ...
Bashar Haddad's user avatar
4 votes
Accepted

change detection

Consider how an algorithm might detect a change. You're observing instances of some random variable, $X_1,X_2,\dots,X_{k-1}$. Suddenly (and unknown to you) at $X_k$ something about the distribution of ...
Quasar's user avatar
  • 156
4 votes

Tools for automatic anomaly detection on a SQL table?

If you need SQL code that runs various outlier detection methods against any arbitrary table, check out my series of articles and code samples geared towards SQL Server. I provide some preliminary ...
SQLServerSteve's user avatar
4 votes

What are some good sources to learn fraud/anomaly detection in normal/time-series data?

Chandola et al's survey is probably the best and most-widely cited survey in the anomaly detection field.
ximiki's user avatar
  • 933
4 votes

Which outlier detection can detect these outliers?

You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor. ...
Marmite Bomber's user avatar
4 votes
Accepted

Anomoly detection method selection

Without putting in the time to look through Azure's documentation, my guess is that their PCA method is really just a way to do a feature reduction, then use some algorithm they have to classify. Best ...
Hobbes's user avatar
  • 1,449
4 votes

Detecting anomalies with neural network

I think that heavily depends on the nature of your data (categorical/continuous). I'd start with simple methods first. Those come to my mind: You can compare distribution of each variable either by ...
HonzaB's user avatar
  • 1,669
4 votes
Accepted

Autoencoder for anomaly detection from feature vectors

Does that mean that my model (or indeed my approach of using an AE) is ineffective Well, it depends. Auto Encoder are a quite broad field, there are many hyperparameters to tune, width, depth, loss ...
RyanMcFlames's user avatar
4 votes

To detect unauthorized access using outlier detection

This question is quite broad. I'll try to set you on the right path, more so than providing a truly complete answer. Theoretical background As others have mentioned, the task you're trying to do is ...
loopbackbee's user avatar
4 votes

Multi Class + Negative Class Image Classification Strategies

There is no general strategy to anything in machine learning. Moreover, research is scattered over multiple domains so it becomes harder to get your head around, "the best strategy for a case". You ...
Mankind_2000's user avatar
4 votes

Why use Variational Autoencoders VAE instead of Autoencoders AE in Anomaly Detection?

Variational autoencoders encourage the model to generalize features and reconstruct images as an aggregation of those features. This is what the latent space encodes, a compressed feature vector. ...
neurokinetikz's user avatar
4 votes

Methods to detect this kind of outliers

I recently had a similar problem (removing abnormal peaks from a time series). That's what I suggest you: Get the smoothed trend. There are several techniques you can employ, such as various forms of ...
Leevo's user avatar
  • 6,255
4 votes

Best anomaly detection algorithm based on two conditions

Since your data is one-dimensional and numeric, I don't think you need any fancy clustering technique. Clustering is useful when your data points have multiple attributes. When there is just a ...
zachdj's user avatar
  • 2,724
4 votes
Accepted

How to detect anomalies in web log data

Since you've used the word 'Automatically', I assume that you're looking for an unsupervised method. Unsupervised Anomaly Detection techniques do not need training data. They are based on two basic ...
Amirhossein Rezaei's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible