Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

150 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5 votes
2 answers
81 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
user avatar
5 votes
1 answer
695 views

Time Series pattern recognition and classification problem

I have some labeled sensor data. Now, I would like to know how to extract features from time series using DFT, DWT, and HAAR transforms. I know that the transformations above transform a signal to ...
user avatar
5 votes
2 answers
285 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
user avatar
4 votes
1 answer
71 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
user avatar
  • 585
3 votes
0 answers
30 views

High dimensional data stream summarization and processing

Can anyone recommend a method for summarizing and processing high dimensional data streams efficiently and effectively for anomaly detection? In fact, I investigated the different methods for data ...
user avatar
  • 51
3 votes
0 answers
4k views

Anomaly detection using k-means clustering in Python

I'm working on an anomaly detection task in Python. Datasets regard a collection of time series coming from a sensor, so data are timestamps and the relative values. In order to find anomalies, I'm ...
user avatar
  • 325
3 votes
0 answers
370 views

Anomaly detection in cooling process data without exact labels

I have a data set where I look at the cooling of a process. The starting temperature may vary between 580 and 180 degrees. I know that at some point the cooling system failed (see examples in the plot)...
user avatar
3 votes
0 answers
569 views

Autoencoder behavior with All White/Black MNIST

I am using a stock auto-encoder anomaly detector from Deeplearning4j. I was getting unexpected results from my own variant of the auto-encoder, which looks for anomalies in my own (non-image) data, ...
user avatar
  • 43
3 votes
1 answer
1k views

IsolationForest Decision Function vs. Anomaly Prediction Question

I'm currently working on an unsupervised anomaly detection project, and for it I'm using IsolationForest through scikit-learn. My question is, why/how is it possible for the model to predict something ...
user avatar
  • 41
3 votes
0 answers
167 views

Clustering with Replicator Neural Network

I'm trying to cluster an unknown set of data with a replicator neural network. The number of clusters is determined by the number of neuron units in the middle layer, multiplied by the number of steps ...
user avatar
  • 275
2 votes
0 answers
91 views

Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?

I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
user avatar
2 votes
1 answer
38 views

Various models giving 99% accuracy for KDDcup 99 dataset for Intrusion Detection, is there some sort of data leak I am missing?

Student who is quite new to all this here. I am currently working with the KDDcup 99 data for intrusion detection using various ML models (and ANN). My problem is that I am getting 99% often for ...
user avatar
2 votes
2 answers
101 views

Log analysis dataset with labeled cybersecurity issues

I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
user avatar
  • 21
2 votes
1 answer
54 views

What's the best way to validate a rare event detection model during training?

When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early ...
user avatar
  • 51
2 votes
1 answer
30 views

Types of artificial anomalies

I am working on some algorithms for anomaly detection The dataset is clean our anomalies so I want to add some artificial anomalies. I have added some anomalies. I get the maximum value of the ...
user avatar
  • 585
2 votes
2 answers
3k views

What does the classification report interpret? Class 1 indicates abnormal data

How to interpret the report and How is precision, recall values are calculated for individual class labels. What is the significance of macro avg ? Does this report signify a good predictions by the ...
user avatar
  • 21
2 votes
1 answer
319 views

Anomaly detection - relation between thresholds and anomalies

I'm developing an anomaly detection program in Python. Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day. Then, using thresholds, find ...
user avatar
  • 325
2 votes
1 answer
14 views

Parameter Adjustment based only on tagged predictions

not sure that this is the best place to post this but if not, please let me know if there is a better stack community. I have an anomaly detection method which has some parameters. I have some data ...
user avatar
2 votes
0 answers
20 views

Temporal outlier Analysis on sensor data

I am working to find anomaly/outliers in sensor data using unsupervised machine learning (without training dataset). I have around 20000 samples taken per minute of various sensors. I just need to ...
user avatar
  • 21
2 votes
3 answers
779 views

Is it meaningful to use word2vec for non-string inputs like time series analysis?

I am working on a project that detects anomalies in a time series. I wonder if I can use word2vec for anomaly detection for non-string inputs like exchange rates?
user avatar
  • 21
2 votes
0 answers
121 views

Can autoencoders take time series into account?

Here, I read the following: The first key to understanding is that HTM relies on data that streams over time (...) By contrast, conventional deep learning uses static data and is therefore time ...
user avatar
  • 480
2 votes
0 answers
57 views

unsupervised outliers detection - possible solution?

I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $ The goal is to identify anomalies in transactions data of traders. For example to ...
user avatar
2 votes
0 answers
134 views

Out of stock / Spike in demand prediction

The goal is to predict out-of-stock situations, either quantitatively (the gap) or qualitatively (out-of-stock likely to happen in next few weeks). Background: We have existing demand planning ...
user avatar
  • 121
2 votes
0 answers
365 views

Sequence classification using oneClass SVM

In the code below, I'm using a sequence to sequence approach as a prediction model for anomaly detection. The data set I'm working with is ADFA-LD. The training phase is done using only normal ...
user avatar
  • 574
2 votes
0 answers
297 views

Comparison between approaches for timeseries anomaly detection

After various days of research, I could take a global picture of the existing methods to perform anomaly detection on time series, namely: Forecasting with Deep Learning. Eg. RADM or LSTM model ...
user avatar
  • 143
2 votes
2 answers
1k views

How can I find anomalies in each row of data?

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility. ...
user avatar
2 votes
1 answer
163 views

Dynamic clustering

I am performing anomaly detection on different datasets and thought to first cluster the dataset and submit each of the clusters to different AD models. I am using HDBSCAN, and in my test dataset I ...
user avatar
  • 61
2 votes
3 answers
1k views

Detect the time at which deviation occurs in time series data

I working on multivariate time series data. I have sensor data generated by a machine every time it is operated. Data set consists of machine_ID(machines of same model), hours_ operated, measurements ...
user avatar
2 votes
0 answers
1k views

How does Elastic's Prelert (formerly Splunk Anomaly Detective App) work?

Background: In recent months, Elastic has purchased Prelert and will actively incorporate it into the Elastic stack (and also discontinue the Splunk Anomaly Detective App!). I am trying to understand ...
user avatar
  • 933
2 votes
1 answer
85 views

Accept any suggestion to create training data from correlation matrix to find odd one out to identify difference in variation

I have N time varying feature vectors obtained by recording different parameters over time.This results in N*N similarity matrix which contains one to one correlations value for each feature. We need ...
user avatar
2 votes
2 answers
106 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
user avatar
  • 585
1 vote
0 answers
14 views

Multiple Timeseries Anomaly Detection - identifying which feature is anomalous?

I have a multivariate time series where I have features such as: temperature set point energy used relative humidity, etc. Currently, I'm creating univariate anomaly detection models in Python using ...
user avatar
1 vote
1 answer
87 views

how to set threshold for anomaly detection

I read one research paper and they said that they are using a threshold for anomaly detection. The threshold is determined to make some proportion of data of the validation dataset labeled as ...
user avatar
  • 175
1 vote
0 answers
18 views

Fraud detection with machine learning with partialy labeled data

I am facing difficulties handling an issue with my dataset. The purpose of my analysis is to find an electricity thief given a dataset. The first dataset: contains the list of all customers (unlabeled,...
user avatar
  • 11
1 vote
1 answer
27 views

Anomaly detection for varying dictionary

I want to detect the anomaly in the processes taking up the most CPU percent. I receive the data as a time series of dictionary values like so: ...
user avatar
  • 11
1 vote
0 answers
60 views

Nonparametric Outlier Detection

Which Nonparametric outlier detection do you suggest to detect outliers (red points) in these plots? I have tested std, IQR, etc., but no good result. It is just one vector including normal and ...
user avatar
  • 443
1 vote
2 answers
66 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
user avatar
  • 1,526
1 vote
0 answers
143 views

Using STL(Seasonal-Trend decomposition using LOESS) for Anomaly detection

I am using STL to decompose my time series data in Season, trend and residual and then by applying this(see below) on residual. I am detecting the anomaly ...
user avatar
1 vote
1 answer
46 views

How to test unsupervised learning methods for anomaly detection?

How to test unsupervised learning methods for anomaly detection? I am looking for a test strategy to evaluate my result of my anomaly detection technique? what is your offer more than evaluate with ...
user avatar
  • 1,526
1 vote
0 answers
19 views

What is the best feature extraction technique for text domain novelty / anomaly detection?

I am working with a text classification system. Here, my data-set has around 30 intents. But the problem is I have no system developed to handle inputs that don't go under any of the intents. So, in ...
user avatar
  • 111
1 vote
0 answers
35 views

One-class SVM formula

Recently I have been studying one-class SVM and am a little bit confused about the offset $\rho$. The common optimization problem is to find a function $f(x)= w^\top x-\rho$ by solving $$\begin{array}...
user avatar
1 vote
1 answer
48 views

An Unsupervised learning method suitable for large categorical data sets

I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
user avatar
  • 11
1 vote
0 answers
128 views

Anomaly detection for high dimensional categorical data

I have a dataset with around 200+ categorical variables $X_i$ and the sizes of their domain $|X_i|$ range from 2 to 8k. So, if I one-hot encode the combination of these variables, the vector space (...
user avatar
  • 11
1 vote
0 answers
101 views

Interpreting one-class SVM

I am new to SVM (one-class) and was practically investigating it. Got some weird result that I can not explain. Let me demonstrate by some small reproducible code and visualization: ...
user avatar
1 vote
0 answers
96 views

How to calculate MAE and threshold in a multivariate time series

I'm trying to understand how to calculate the MAE in my time series and then the thresholds to understand which of my data in the test set are anomalies. I'm following this tutorial, which is based on ...
user avatar
  • 43
1 vote
1 answer
49 views

Labels as features in anomaly detection

I have a dataset born to solve a classification problem. Due to the imbalances of the Y, i choose to move to an anomaly detection task. Should I use the Y i have inside the anomaly detection model as ...
user avatar
  • 11
1 vote
1 answer
246 views

K-Prototype for anomaly detection

I have logs of the form (e.g. from a gym login).. the representational case is so: UserName, Login time, timeSpend_on_weights, time_spent_on_elliptical ...
user avatar
1 vote
1 answer
45 views

Multivariate Gaussian distribution - Covariance vs linear dependence

From prof. Andrew Ng's Multivariate Gaussian distribution lecture, covariance measures linear dependency between features, in which case we might use Multivariate Gaussian distribution with covariance ...
user avatar
  • 145
1 vote
0 answers
269 views

Hyperparameter tuning one-class svm

I have a problem where I am trying to apply a one-class svm to detect outliers. I am training on a dataset of true cases using a one-class radial svm and then predicting for both false and true cases. ...
user avatar
1 vote
0 answers
16 views

Reporting false alarm rate and detection rate in shilling attack detection without given any labels

I have a dataset of ratings given to movies by users. I've applied the method mentioned in this paper to detect fake votes(I've used $H(X)$ and $M(X)$ measures). But the dataset I'm using doesn't have ...
user avatar