Questions tagged [anomaly-detection]
Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.
352
questions
0
votes
0
answers
44
views
Modelling LSTM Autoencoder for anomaly detection with multiple Time Series
I'm currently working on LSTM autoencoder for anomaly detection. My main problem is I have multiple time series - each individual time series corresponds to a different customer, detailing their sales ...
- 1
1
vote
0
answers
123
views
How can I apply NLP/NLU methods for anomaly detection in structured log data?
I have a dataset of logs with a specific structured format, and I'm looking for the best approach to detect anomalies within this data. I've already experimented with autoencoders and clustering ...
- 11
0
votes
0
answers
18
views
Detecting Anomalies Using LSTMs
I'm studying this article. The authors used a two-step approach to detect anomalies. First, they used an LSTM to learn the normal behavior of the data. Then, they used the dynamic error thresholds to ...
- 113
0
votes
0
answers
15
views
Unsupervised rule extraction of categorical data
I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows.
I am able to classify a ...
0
votes
1
answer
76
views
Constructing an LSTM autoencoder for variable-lentgth sequences
I would like to construct an LSTM autoencoder model for sequence anomaly detection where the sequences can be varying in length. I understand based on this answer that padding and masking can be used ...
- 123
1
vote
1
answer
72
views
How to Justify Anomalies Detected by Unsupervised Anomaly Detection Models? [closed]
I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several ...
0
votes
1
answer
63
views
Use computer vision to detect door blockage
I want to detect door blockage on a camera. Basically if the exit door is blocked by an object, it detects it as an anomaly. How can we do it? Is it possible to do it using OpenCV?
Remember, it doesn’...
- 217
0
votes
1
answer
21
views
Human Classification Error Detection
I'm working on a model to detect errors on human classifications.
I already have a classifier model M: X -> Y, but I need now a model M' (X,Y) -> {0,1}.
Y contains a lot of classes (~5-6k).
My ...
- 323
0
votes
0
answers
12
views
How to build a recommender system for recommending skewed positive class examples using e.g. one-class classification model or anomaly detection?
I want to build a recommendation system for accounts purchasing some items. The ratio of purchase events to view events is very low (less than 1% items that are viewed get bought). Right now I am ...
- 251
0
votes
0
answers
37
views
What is the best way to approach anomaly detection on a data set using machine learning?
I am looking to help on where to start exploring machine learning when it comes to data processing.
Say I have the following csv file with hundreds of thousands of rows of data:
ID
Amount
Overdue (...
- 1
0
votes
0
answers
18
views
What is the exctual purpose of a real-time log anomaly detection?
I'm sorry for asking a bit outlier topic question.
Currently, I'm trying to understand log parser processing and log analysis for anomaly detection in the ML/DL class. There are so many references for ...
- 113
0
votes
1
answer
97
views
Anomaly Detection: Large number of categories
Looking for some advice.
I am working on an Anomaly detection problem, I am looking at parcels being transported from A-B and want to identify which parcels are considered anomalies for given routes.
...
- 1
0
votes
1
answer
24
views
Best practice labeling grouped anomalies for object detection
I would like to train object detection model (e.g. YOLO) for images that contain anomalies. The anomalies are essentially the holes in a surface of different sizes. How do I label correctly such ...
- 3
0
votes
0
answers
18
views
AUC drops below 0.5 even though dataset stays similar
I'm programming an anomaly detection on a given dataset:
Toyadmos dataset: https://arxiv.org/abs/1908.03299
Of this dataset, I'm investigating the ToyCar data, which has '4 cases':
(quote)"Each “...
0
votes
0
answers
64
views
How to use IsolationForest for anomaly detection of a 1D-array, given given another feature?
[Beginner at ML]
Hi,
I would like to use unsupervised anomaly detection of spectrograms.
Currently, I am trying IsolationForest on a bunch of 1D-arrays (light intensity vs wavelength) in order to ...
- 1
0
votes
0
answers
24
views
Varying feature vector lengths for learning
[I am a total beginner in machine learning algorithms]
I have 10 spectrograms (lines) for phytoplankton (each composed of 288 points).
Each spectrogram is associated with a phytoplankton dendity data ...
- 1
0
votes
0
answers
26
views
Anomaly detection on subset of categorical data
Imagine that I'm a health department, and I have a data stream that describes whether someone gets sick after eating food from a restaurant. Each data point in the data stream arrives 24 hours after ...
- 101
0
votes
0
answers
77
views
Performance metrics for outlier/anomaly detection
I am currently seeking for metrics i can use to evaluate a model for outlier/anomaly detection without ground truth. The only thing i came up with for now is to use the scores/probas returned by my ...
- 1
0
votes
1
answer
10
views
RCA on top of existing anomalies
When we work with timeseries data containing multiple features eg. a sensor data. we can detect anomalies using cluster, supervised and semi supervised based approaches (Eg. Isolation, Autoencoder Etc)...
- 111
0
votes
1
answer
43
views
Unsupervised anomaly detection - dataset with multiple users
I'm confused on how to go about an issue.
I'm trying to implement an unsupervised model, using a dataset that is essentially a log file. This dataset contains a variety of features, but most ...
- 1
1
vote
0
answers
64
views
Threshold for auto encoder anomaly detection
I have fitted an auto encoder on 25-dimensional time series data hoping to be able to detect anomalies. training set is 100k observations, testset for threshold setting is 10k observations. all ...
- 21
0
votes
0
answers
28
views
Cluster-Based Anomaly Detection (and PCA)
I have a dataset of user operations (250 types of operations) in a trading platform and my task is to extract features, flagging rules, other insight for fraud prevention/anomaly detection, or some ...
- 101
1
vote
1
answer
37
views
Best way to detect newly incoming anomalies in two timeseries?
I have two devices that both send data (let's say temperature). I need to be able to detect if one of the devices reports an unusual/anomalous reading. The case if both of the devices report a "...
- 13
2
votes
1
answer
187
views
Early anomaly detection / Failure prediction on time series
My problem here is that I want to predict failures in advance with respect to their occurrence. I have sensors mounted on my machine and with a certain frequency, they send data to my database. ...
- 45
1
vote
1
answer
20
views
Underfitting and perfomance metrics in unsupervised methods
My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
- 11
-2
votes
1
answer
97
views
How to increase , precision-recall value in your Deep learning model
I am getting good accuracy metrics around 80 with precision =66, recall =37, F1 =47. How can I improve precision, and recall metrics in anomaly detection scenarios.. any suggestions?
- 171
1
vote
1
answer
99
views
Determining threshold for KMeans anomaly detection
I'm trying to use KMeans for anomaly detection, and I know that a threshold is needed to determine the anomalies. I've seen many articles talking about how to choose K, but none talks about how to ...
- 131
0
votes
0
answers
19
views
Determine unusual occurrence of words in classes
I am working on a project where I have 20+ classes/groups. Each of these groups perform certain text searches. I am looking for specific keywords example 'code' which is an anomaly. The challenge is ...
- 11
0
votes
0
answers
27
views
How to detect Novelty from different ranges of target variable?
I've a dataset of multiple categorical columns along with a target column that is continuous. Assume combination of categorical columns has a different range of values of target. Ex
Col1 - col2 - col3 ...
0
votes
0
answers
36
views
Video anomaly detection/ Evaluation AUC
I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array.
scores = (scores - min(scores))/max(...
- 1
0
votes
1
answer
35
views
Sound Anomaly Detection
What is the recommended directory structure for sound anomaly detection using Keras CNN (Unsupervised) ?
After converting the sound files into spectrograms. Code examples will be highly appreciated.
- 1
0
votes
1
answer
35
views
detecting abnormality in a specific feature with respect to others (unsupervised?)
I have a large dataset with a feature y which is dependent in part on features x1 and x2. All features are noisy, and y is also dependent on other parameters not captured in the dataset. I would like ...
0
votes
0
answers
14
views
What is the use of validation dataset when doing regression-based outlier detection?
I have a dataset where data are velocity data splitted as:
60%(train - non-anomalous)
20%(validation - 50% of it anomalous)
20%(test - 50% of it anomalous).
From my understanding, when doing outlier ...
- 131
4
votes
1
answer
474
views
How to detect anomalies in web log data
I have the following challenge: We have the web logs of a platform where people can download publications and we need to detect anomalies.
From time to time and only by chance we observe spikes in ...
- 143
0
votes
0
answers
15
views
Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?
I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case?
The system I am building assigns classes (-1, 0, +1) to observations ...
0
votes
2
answers
326
views
Time Series - Anomaly Detection
I have time-series data with alerts (every minute) that I need to find anomalies in.
I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data.
Which ...
- 101
0
votes
0
answers
26
views
Handling features dependent on a data field that may not be present in sample
I'm trying to build an anomaly detection model using Isolation Forest. I currently have 12 features, about half of them depends on the presence of a particular data field, say ...
- 131
0
votes
0
answers
34
views
Why anomaly detection with IForest does not give expected results?
My goal is to develop an anomaly detection model with Isolation Forest in order to distinguish between normal and anomalous IPs by analyzing the web access logs and ...
- 101
1
vote
2
answers
57
views
Validate Unsupervised Binary Classification
I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
- 11
0
votes
0
answers
14
views
Find the most impactfuls parameters multivariate output unsupervised ML
I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
- 1
1
vote
0
answers
52
views
How to find inconsistency in data?
I wonder how I can find any inconsistencies in data. To show what I mean let's consider the following example:
Imagine I have a problem of finding out why the speed (S) of the vehicle is low. I have ...
- 121
1
vote
1
answer
46
views
How can I find anomalies in features based on difference between true and predicted targets?
Generally the problem is the following : there is target (efficiency of mobile stations). The goal is to find stations which underperform and highlight the reasons of that. Other than this it is ...
- 121
0
votes
2
answers
83
views
Anomaly prediction/forecasting in timeseries?
What options exist in order to forecast when next observation will be an outlier in a time series? Initially, I thought to train a simple forecasting model, which turned out to decently predict the ...
- 1
0
votes
0
answers
34
views
Understanding Isolation Forest predictions
I'm running sklearn's IsolationForest on a dataset containing 2 classes of data, one that I know is the anomaly (~1.5% of the entire dataset), the other is the normal dataset.
I'm using this (shuffled)...
- 131
0
votes
0
answers
26
views
Dealing with conditional variables in anomaly detection
I am working on an anomaly detection model and I have a conditional variable, i.e., it is zero or it has an amount like below histogram. Suppose the variable shows the time when a machine is not ...
- 11
0
votes
1
answer
320
views
How to fit a model on validation_data?
can you help me understand this better? I need to detect anomalies so I am trying to fit an lstm model using validation_data but the losses does not converge. Do they really need to converge? Does the ...
- 13
0
votes
1
answer
128
views
Cross-validation for anomaly detection on time series data
I want to perform k-fold cross-validation for the setting where I have a training dataset consisting of a sequential time series that is fully benign and a test dataset (also a sequential time series)...
1
vote
1
answer
140
views
is it good to have 100% accuracy on validation?
i'm still new in machine learning. currently i'm creating an anomaly detection for flight data. it is a multivariate time series data that include timestamp, latitude, longitude, velocity and altitude ...
- 13
1
vote
1
answer
205
views
How to compute threshold?
I would like to detect anomalies for univariate time series data. Most examples on internet show that, after you predict the model, you calculate a threshold for the training data and a MAE test loss ...
- 13
0
votes
1
answer
90
views
An autoencoder setup for anomaly detection
I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
- 11