Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
42 views

Modelling LSTM Autoencoder for anomaly detection with multiple Time Series

I'm currently working on LSTM autoencoder for anomaly detection. My main problem is I have multiple time series - each individual time series corresponds to a different customer, detailing their sales ...
suziex's user avatar
  • 1
1 vote
0 answers
121 views

How can I apply NLP/NLU methods for anomaly detection in structured log data?

I have a dataset of logs with a specific structured format, and I'm looking for the best approach to detect anomalies within this data. I've already experimented with autoencoders and clustering ...
Coolfrog's user avatar
0 votes
0 answers
18 views

Detecting Anomalies Using LSTMs

I'm studying this article. The authors used a two-step approach to detect anomalies. First, they used an LSTM to learn the normal behavior of the data. Then, they used the dynamic error thresholds to ...
Vahid Shams's user avatar
0 votes
0 answers
15 views

Unsupervised rule extraction of categorical data

I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows. I am able to classify a ...
user153170's user avatar
0 votes
1 answer
71 views

Constructing an LSTM autoencoder for variable-lentgth sequences

I would like to construct an LSTM autoencoder model for sequence anomaly detection where the sequences can be varying in length. I understand based on this answer that padding and masking can be used ...
interoception's user avatar
1 vote
1 answer
69 views

How to Justify Anomalies Detected by Unsupervised Anomaly Detection Models? [closed]

I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several ...
Jais Varghese Joseph's user avatar
0 votes
1 answer
60 views

Use computer vision to detect door blockage

I want to detect door blockage on a camera. Basically if the exit door is blocked by an object, it detects it as an anomaly. How can we do it? Is it possible to do it using OpenCV? Remember, it doesn’...
Tina J's user avatar
  • 217
0 votes
1 answer
21 views

Human Classification Error Detection

I'm working on a model to detect errors on human classifications. I already have a classifier model M: X -> Y, but I need now a model M' (X,Y) -> {0,1}. Y contains a lot of classes (~5-6k). My ...
EzrielS's user avatar
  • 323
0 votes
0 answers
12 views

How to build a recommender system for recommending skewed positive class examples using e.g. one-class classification model or anomaly detection?

I want to build a recommendation system for accounts purchasing some items. The ratio of purchase events to view events is very low (less than 1% items that are viewed get bought). Right now I am ...
exAres's user avatar
  • 251
0 votes
0 answers
37 views

What is the best way to approach anomaly detection on a data set using machine learning?

I am looking to help on where to start exploring machine learning when it comes to data processing. Say I have the following csv file with hundreds of thousands of rows of data: ID Amount Overdue (...
Doug's user avatar
  • 1
0 votes
0 answers
17 views

What is the exctual purpose of a real-time log anomaly detection?

I'm sorry for asking a bit outlier topic question. Currently, I'm trying to understand log parser processing and log analysis for anomaly detection in the ML/DL class. There are so many references for ...
orde.r's user avatar
  • 113
0 votes
1 answer
92 views

Anomaly Detection: Large number of categories

Looking for some advice. I am working on an Anomaly detection problem, I am looking at parcels being transported from A-B and want to identify which parcels are considered anomalies for given routes. ...
darren's user avatar
  • 1
0 votes
1 answer
23 views

Best practice labeling grouped anomalies for object detection

I would like to train object detection model (e.g. YOLO) for images that contain anomalies. The anomalies are essentially the holes in a surface of different sizes. How do I label correctly such ...
In777's user avatar
  • 3
0 votes
0 answers
18 views

AUC drops below 0.5 even though dataset stays similar

I'm programming an anomaly detection on a given dataset: Toyadmos dataset: https://arxiv.org/abs/1908.03299 Of this dataset, I'm investigating the ToyCar data, which has '4 cases': (quote)"Each “...
Brecht De Cock's user avatar
0 votes
0 answers
62 views

How to use IsolationForest for anomaly detection of a 1D-array, given given another feature?

[Beginner at ML] Hi, I would like to use unsupervised anomaly detection of spectrograms. Currently, I am trying IsolationForest on a bunch of 1D-arrays (light intensity vs wavelength) in order to ...
Marianne's user avatar
0 votes
0 answers
24 views

Varying feature vector lengths for learning

[I am a total beginner in machine learning algorithms] I have 10 spectrograms (lines) for phytoplankton (each composed of 288 points). Each spectrogram is associated with a phytoplankton dendity data ...
Marianne's user avatar
0 votes
0 answers
25 views

Anomaly detection on subset of categorical data

Imagine that I'm a health department, and I have a data stream that describes whether someone gets sick after eating food from a restaurant. Each data point in the data stream arrives 24 hours after ...
leontp587's user avatar
  • 101
0 votes
0 answers
76 views

Performance metrics for outlier/anomaly detection

I am currently seeking for metrics i can use to evaluate a model for outlier/anomaly detection without ground truth. The only thing i came up with for now is to use the scores/probas returned by my ...
Raphael's user avatar
0 votes
1 answer
10 views

RCA on top of existing anomalies

When we work with timeseries data containing multiple features eg. a sensor data. we can detect anomalies using cluster, supervised and semi supervised based approaches (Eg. Isolation, Autoencoder Etc)...
duck's user avatar
  • 111
0 votes
1 answer
42 views

Unsupervised anomaly detection - dataset with multiple users

I'm confused on how to go about an issue. I'm trying to implement an unsupervised model, using a dataset that is essentially a log file. This dataset contains a variety of features, but most ...
Kurt's user avatar
  • 1
1 vote
0 answers
64 views

Threshold for auto encoder anomaly detection

I have fitted an auto encoder on 25-dimensional time series data hoping to be able to detect anomalies. training set is 100k observations, testset for threshold setting is 10k observations. all ...
Student's user avatar
  • 21
0 votes
0 answers
28 views

Cluster-Based Anomaly Detection (and PCA)

I have a dataset of user operations (250 types of operations) in a trading platform and my task is to extract features, flagging rules, other insight for fraud prevention/anomaly detection, or some ...
marcustulliuscicero's user avatar
1 vote
1 answer
36 views

Best way to detect newly incoming anomalies in two timeseries?

I have two devices that both send data (let's say temperature). I need to be able to detect if one of the devices reports an unusual/anomalous reading. The case if both of the devices report a "...
Jamess11's user avatar
2 votes
1 answer
179 views

Early anomaly detection / Failure prediction on time series

My problem here is that I want to predict failures in advance with respect to their occurrence. I have sensors mounted on my machine and with a certain frequency, they send data to my database. ...
Andrea's user avatar
  • 45
1 vote
1 answer
20 views

Underfitting and perfomance metrics in unsupervised methods

My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
BilboBuggins's user avatar
-2 votes
1 answer
91 views

How to increase , precision-recall value in your Deep learning model

I am getting good accuracy metrics around 80 with precision =66, recall =37, F1 =47. How can I improve precision, and recall metrics in anomaly detection scenarios.. any suggestions?
user12's user avatar
  • 171
1 vote
1 answer
97 views

Determining threshold for KMeans anomaly detection

I'm trying to use KMeans for anomaly detection, and I know that a threshold is needed to determine the anomalies. I've seen many articles talking about how to choose K, but none talks about how to ...
Rayne's user avatar
  • 131
0 votes
0 answers
19 views

Determine unusual occurrence of words in classes

I am working on a project where I have 20+ classes/groups. Each of these groups perform certain text searches. I am looking for specific keywords example 'code' which is an anomaly. The challenge is ...
kruparulz14's user avatar
0 votes
0 answers
27 views

How to detect Novelty from different ranges of target variable?

I've a dataset of multiple categorical columns along with a target column that is continuous. Assume combination of categorical columns has a different range of values of target. Ex Col1 - col2 - col3 ...
soumalya saha's user avatar
0 votes
0 answers
35 views

Video anomaly detection/ Evaluation AUC

I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array. scores = (scores - min(scores))/max(...
TecK97's user avatar
  • 1
0 votes
1 answer
35 views

Sound Anomaly Detection

What is the recommended directory structure for sound anomaly detection using Keras CNN (Unsupervised) ? After converting the sound files into spectrograms. Code examples will be highly appreciated.
Cactus's user avatar
  • 1
0 votes
1 answer
35 views

detecting abnormality in a specific feature with respect to others (unsupervised?)

I have a large dataset with a feature y which is dependent in part on features x1 and x2. All features are noisy, and y is also dependent on other parameters not captured in the dataset. I would like ...
user18236139's user avatar
0 votes
0 answers
14 views

What is the use of validation dataset when doing regression-based outlier detection?

I have a dataset where data are velocity data splitted as: 60%(train - non-anomalous) 20%(validation - 50% of it anomalous) 20%(test - 50% of it anomalous). From my understanding, when doing outlier ...
Mr. Panda's user avatar
  • 131
4 votes
1 answer
474 views

How to detect anomalies in web log data

I have the following challenge: We have the web logs of a platform where people can download publications and we need to detect anomalies. From time to time and only by chance we observe spikes in ...
jfix's user avatar
  • 143
0 votes
0 answers
15 views

Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?

I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case? The system I am building assigns classes (-1, 0, +1) to observations ...
HackJob99's user avatar
0 votes
2 answers
320 views

Time Series - Anomaly Detection

I have time-series data with alerts (every minute) that I need to find anomalies in. I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data. Which ...
Jilaba Hindga's user avatar
0 votes
0 answers
26 views

Handling features dependent on a data field that may not be present in sample

I'm trying to build an anomaly detection model using Isolation Forest. I currently have 12 features, about half of them depends on the presence of a particular data field, say ...
Rayne's user avatar
  • 131
0 votes
0 answers
34 views

Why anomaly detection with IForest does not give expected results?

My goal is to develop an anomaly detection model with Isolation Forest in order to distinguish between normal and anomalous IPs by analyzing the web access logs and ...
Kosmylo's user avatar
  • 101
1 vote
2 answers
55 views

Validate Unsupervised Binary Classification

I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
fpialcoi_o's user avatar
0 votes
0 answers
14 views

Find the most impactfuls parameters multivariate output unsupervised ML

I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
Art's user avatar
  • 1
1 vote
0 answers
51 views

How to find inconsistency in data?

I wonder how I can find any inconsistencies in data. To show what I mean let's consider the following example: Imagine I have a problem of finding out why the speed (S) of the vehicle is low. I have ...
Nourless's user avatar
  • 121
1 vote
1 answer
46 views

How can I find anomalies in features based on difference between true and predicted targets?

Generally the problem is the following : there is target (efficiency of mobile stations). The goal is to find stations which underperform and highlight the reasons of that. Other than this it is ...
Nourless's user avatar
  • 121
0 votes
2 answers
82 views

Anomaly prediction/forecasting in timeseries?

What options exist in order to forecast when next observation will be an outlier in a time series? Initially, I thought to train a simple forecasting model, which turned out to decently predict the ...
M Reyes's user avatar
0 votes
0 answers
34 views

Understanding Isolation Forest predictions

I'm running sklearn's IsolationForest on a dataset containing 2 classes of data, one that I know is the anomaly (~1.5% of the entire dataset), the other is the normal dataset. I'm using this (shuffled)...
Rayne's user avatar
  • 131
0 votes
0 answers
26 views

Dealing with conditional variables in anomaly detection

I am working on an anomaly detection model and I have a conditional variable, i.e., it is zero or it has an amount like below histogram. Suppose the variable shows the time when a machine is not ...
8Simon8's user avatar
  • 11
0 votes
1 answer
318 views

How to fit a model on validation_data?

can you help me understand this better? I need to detect anomalies so I am trying to fit an lstm model using validation_data but the losses does not converge. Do they really need to converge? Does the ...
warriorforce's user avatar
0 votes
1 answer
126 views

Cross-validation for anomaly detection on time series data

I want to perform k-fold cross-validation for the setting where I have a training dataset consisting of a sequential time series that is fully benign and a test dataset (also a sequential time series)...
kohlstein's user avatar
1 vote
1 answer
140 views

is it good to have 100% accuracy on validation?

i'm still new in machine learning. currently i'm creating an anomaly detection for flight data. it is a multivariate time series data that include timestamp, latitude, longitude, velocity and altitude ...
farhanrbn's user avatar
1 vote
1 answer
204 views

How to compute threshold?

I would like to detect anomalies for univariate time series data. Most examples on internet show that, after you predict the model, you calculate a threshold for the training data and a MAE test loss ...
warriorforce's user avatar
0 votes
1 answer
87 views

An autoencoder setup for anomaly detection

I am doing anomaly detection using machine learning. i have tried different models like isolation forest, SVM and KNN. The maximum accuracy that I can get from each of them is $80\%$ accordind to my ...
Riva11's user avatar
  • 11

1
2 3 4 5
8