Questions tagged [anomaly-detection]
Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.
359
questions
0
votes
0
answers
9
views
What are the most important evaluation metrics for anomaly segmentation?
When people talk about anomaly segmentation models, they often mention evaluation metrics like F1 score, AP, AUROC, and AUPRO. But which one really matters most when comparing models, and why? I'm ...
- 1
7
votes
4
answers
24k
views
Anomaly detection on time series
I've just started working on an anomaly detection development in Python.
My data sets are a collection of timeseries. More in details, data are coming from some sensors/meters which record and ...
- 1
1
vote
0
answers
13
views
Autoencoders are fitting anomalies too good
I have a set of ~ 5000 greyscale images with resolution of 64x128. I want to do an unsupervised anomaly detection. As a first try, I chose convolutional autoencoders (AE) and trained an AE model. I ...
- 11
2
votes
2
answers
167
views
Is it safe to use labels created from unsupervised model to train a supervised model using the same data?
I have a dataset where I have to detect anomalies. Now, I use a subset of the data(let's call that subset A) and apply the DBSCAN algorithm to detect anomalies on set A.Once the anomalies are detected,...
CommunityBot
- 1
0
votes
0
answers
7
views
Efficient anomaly detection in unordered market data - is it possible?
I'm a little bit stuck on how to efficiently model anomaly detection for the following problem, probably because of my lack of experience with time series modelling:
I retrieve market data sorted by ...
- 1
0
votes
0
answers
6
views
How to Identify Equipment Churn from Laboratory Service Records Without Direct Churn Labels?
I'm analyzing a dataset encompassing 20 years of laboratory equipment service records, which includes the equipment ID, service dates, types of equipment (HOOD_TYPE), and descriptions of performed ...
- 1
2
votes
2
answers
5k
views
What does the classification report interpret? Class 1 indicates abnormal data
How to interpret the report and How is precision, recall values are calculated for individual class labels. What is the significance of macro avg ? Does this report signify a good predictions by the ...
CommunityBot
- 1
1
vote
3
answers
170
views
Anomaly detection in nominal big data
I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
CommunityBot
- 1
1
vote
2
answers
182
views
How to find vertical clusters in 1-D data
I have residuals of a multivariate time series data obtained from sensors on a server.spikes in the plots of residuals indicate abnormal server state. I want to cluster the data into vertical clusters ...
CommunityBot
- 1
0
votes
2
answers
183
views
Anomaly detection in a database
We have a production database. The load on the database varies at different times. I want to identify anomalies; for e.g, the number of database processes responding to user queries at 9 am is 100 for ...
CommunityBot
- 1
1
vote
1
answer
175
views
Semi-supervised anomaly detection
I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods.
Now, the thing is, there might ...
CommunityBot
- 1
2
votes
2
answers
279
views
Log analysis dataset with labeled cybersecurity issues
I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
CommunityBot
- 1
1
vote
1
answer
84
views
Get how similarity between the training data and the income data?
I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections
I want something like a semi-supervised model. I want to predict ...
CommunityBot
- 1
1
vote
2
answers
162
views
Anomaly detection for time series data with only positive samples?
I'm having a time series ECG dataset. I want to do anomaly detection (anything different from normal ECG should be abnormal).
The point is I'm having only positive samples with very few negative ...
CommunityBot
- 1
0
votes
1
answer
282
views
Incremental learning on Autoencoder for anomaly detection
I want to incrementally train my pre-trained autoencoder model on data being received every minute. Based on this thread, successive calls to model.fit will incrementally train the model. However, the ...
CommunityBot
- 1
0
votes
2
answers
226
views
Evaluation metric for imbalanced data
Hi I'm a CS graduate student
I have a question for AI or data experts. I'm writing a paper
My dataset is time-series sensor data and anomaly (positive class) ratio is between 5% and 6%
you can see the ...
CommunityBot
- 1
0
votes
1
answer
518
views
How to gps data anomaly detection in python
I have gps format dataset lat, lon. I want to detection anomaly using python.
I tested knn, smv, cof, iforest using pycaret. But i did not.
These colors anomlay because the
angle change is too much
...
CommunityBot
- 1
1
vote
3
answers
145
views
Real-Time Outlier/Anomaly Detection?
My data is the usage/playing statistics for players of a specific game. One data point for a user is aggregated statistics for one week.
The goal is:
to detect when the player's account was stolen/...
CommunityBot
- 1
0
votes
0
answers
15
views
Addressing prolonged high matrix profile values in anomaly detection
In an anomaly detection task, I have a data stream where each new data point is generated every 5 minutes. When a new data point arrives, I compute the matrix profile using Stumpy's stumpi function. ...
- 101
0
votes
1
answer
125
views
Word2vec to encode medical procedures when using isolation forests
I am planning to use Isolation Forests in R (solitude package) to identify outlier medical claims in my data.
Each row of my data represents the group of drugs that each provider has administered in ...
CommunityBot
- 1
0
votes
1
answer
261
views
What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?
I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
CommunityBot
- 1
2
votes
1
answer
96
views
Accept any suggestion to create training data from correlation matrix to find odd one out to identify difference in variation
I have N time varying feature vectors obtained by recording different parameters over time.This results in N*N similarity matrix which contains one to one correlations value for each feature. We need ...
CommunityBot
- 1
1
vote
2
answers
2k
views
how to set threshold for anomaly detection
I read one research paper and they said that they are using a threshold for anomaly detection. The threshold is determined to make some proportion of data of the validation dataset labeled as ...
CommunityBot
- 1
1
vote
1
answer
62
views
Labels as features in anomaly detection
I have a dataset born to solve a classification problem. Due to the imbalances of the Y, i choose to move to an anomaly detection task. Should I use the Y i have inside the anomaly detection model as ...
CommunityBot
- 1
1
vote
1
answer
337
views
Isolation Forest Score Function Theory
I am currently reading this paper on isolation forests. In the section about the score function, they mention the following. For context, $h(x)$ is definded as the path length of a data point ...
CommunityBot
- 1
6
votes
3
answers
324
views
Anomaly detection using clustering of highly correlated Categorical data
My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands ...
CommunityBot
- 1
0
votes
2
answers
408
views
Time Series - Anomaly Detection
I have time-series data with alerts (every minute) that I need to find anomalies in.
I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data.
Which ...
CommunityBot
- 1
4
votes
1
answer
96
views
anomaly detection in vehicle sensor data
I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors.
It seems like there is no proper book or article that goes deeper into ...
CommunityBot
- 1
0
votes
0
answers
42
views
Unsupervised Log Anomaly Detection
I am thinking about using the variational autoencoder model for anomaly detection . I have an Android Logs dataset.
As the logs generated are a representative of time series type of data I thought ...
0
votes
1
answer
355
views
How can I generate anomalies in a dataset?
I'm building a tensorflow model to detect anomalies in an electricity smart meter data
and I'm using UK-DALE Dataset. How can I introduce anomalies in the data so I can test the model?
CommunityBot
- 1
1
vote
1
answer
244
views
Decision trees for anomaly detection
Problem
From what I understand, a common method in anomaly detection consists in building a predictive model trained on non-anomalous training data, and perform anomaly detection using the error of ...
CommunityBot
- 1
0
votes
1
answer
43
views
Anomaly Detection in Log Data using LSTM
Problem Overview:
I am currently working on a project involving anomaly detection in log data. The anomalies are defined by deviations from historical patterns. The log data has a simple structure: [...
- 1,045
5
votes
2
answers
104
views
Detecting abundance of a certain periodic pattern in a time series?
I am really stumped at the moment about how to solve a particular problem. I have many time series like this:
This represents the number of hours a person spends on a website each day throughout the ...
CommunityBot
- 1
0
votes
1
answer
626
views
how to set threshold value by looking at loss distribution in anomaly detection task
I am following this tutorial https://towardsdatascience.com/lstm-autoencoder-for-anomaly-detection-e1f4f2ee7ccf to use LSTM autoencoder to detect anomalies in my unsupervised dataset. they plotted ...
CommunityBot
- 1
2
votes
1
answer
490
views
Anomaly detection - relation between thresholds and anomalies
I'm developing an anomaly detection program in Python.
Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day.
Then, using thresholds, find ...
CommunityBot
- 1
0
votes
1
answer
99
views
is there a way to check if i got a "good price" on something?
I'm looking at some data. Actually, the Boston Housing dataset is probably a good proxy for it: https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html
I'm wondering if there's a way to ...
CommunityBot
- 1
3
votes
1
answer
74
views
What's the best way to validate a rare event detection model during training?
When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early ...
CommunityBot
- 1
2
votes
1
answer
171
views
Various models giving 99% accuracy for KDDcup 99 dataset for Intrusion Detection, is there some sort of data leak I am missing?
Student who is quite new to all this here. I am currently working with the KDDcup 99 data for intrusion detection using various ML models (and ANN). My problem is that I am getting 99% often for ...
CommunityBot
- 1
1
vote
1
answer
101
views
An Unsupervised learning method suitable for large categorical data sets
I want to detect anomalies in the bank data set in an unsupervised learning method. However, in the bank data set, all columns except time and amount were categorical data, and about half of them had ...
CommunityBot
- 1
2
votes
1
answer
356
views
Comparison between approaches for timeseries anomaly detection
After various days of research, I could take a global picture of the existing methods to perform anomaly detection on time series, namely:
Forecasting with Deep Learning. Eg. RADM or LSTM model
...
CommunityBot
- 1
1
vote
2
answers
119
views
unsupervised anomaly detection for univariate fast frequency time series data?
I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer
This time series ...
CommunityBot
- 1
0
votes
1
answer
18
views
Univariate anomaly / outlier detection
I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection.
I have a dataset of around 60K data points. Each data point is part ...
- 1,482
0
votes
1
answer
125
views
Time series forecasting with non-temporal information (exogenous features)
I'm reviewing many time-series algorithms and libraries, such as Prophet, darts, auto_ts, etc.
All libraries discuss univariant time-series (where the task is to forecast based on a single time-series)...
CommunityBot
- 1
1
vote
2
answers
1k
views
Cross-Validation in Anomaly Detection with Labelled Data
I am working on a project where I train anomaly detection algorithms Isolation Forest and Auto-Encoder. My data is labelled so I have the ground truth but the nature of the problem requires ...
2
votes
1
answer
640
views
Can one use PCA to reduce the dimensionality of One-Hot-Encoded data?
I read a couple times that PCA was used as a method to reduce dimensionality for one-hot-encoded data. However, there were also some comments that using PCA is not a good idea since one-hot-encoded ...
CommunityBot
- 1
0
votes
2
answers
2k
views
Percentile as a threshold for Anomaly Detection?
I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
CommunityBot
- 1
2
votes
2
answers
1k
views
How can I find anomalies in each row of data?
I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility.
...
CommunityBot
- 1
1
vote
1
answer
469
views
K-Prototype for anomaly detection
I have logs of the form (e.g. from a gym login).. the representational case is so:
UserName, Login time, timeSpend_on_weights, time_spent_on_elliptical
...
CommunityBot
- 1
1
vote
1
answer
227
views
Algorithm suggestion for anomaly detection in multivariate time series data
I have time series data containing user actions at certain time intervals
eg
...
CommunityBot
- 1
3
votes
1
answer
2k
views
IsolationForest Decision Function vs. Anomaly Prediction Question
I'm currently working on an unsupervised anomaly detection project, and for it I'm using IsolationForest through scikit-learn. My question is, why/how is it possible for the model to predict something ...
CommunityBot
- 1