Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

What to put for X_train, y_train when using it for unsupervised LSTM for anomaly detection?

I have a dataset with 5 features (excluding the date) [Result, Ward, Age, Facility, Resource] . The train dataset has non-anomalous data, and the test dataset will have some anomalous data. This ...
Chiken's user avatar
  • 1
0 votes
0 answers
10 views

User Behaviour Anomaly Detection

I´m trying to detect anomalies in the behaviour of users in an app. My dataset have several fields, but I think the most important ones are User Id, TimeStamp, and Event_name. So for example I could ...
Juan Pablo Pereira's user avatar
0 votes
0 answers
15 views

What are the most important evaluation metrics for anomaly segmentation?

When people talk about anomaly segmentation models, they often mention evaluation metrics like F1 score, AP, AUROC, and AUPRO. But which one really matters most when comparing models, and why? I'm ...
Mosh Geb's user avatar
2 votes
1 answer
32 views

Autoencoders are fitting anomalies too good

I have a set of ~ 5000 greyscale images with resolution of 64x128. I want to do an unsupervised anomaly detection. As a first try, I chose convolutional autoencoders (AE) and trained an AE model. I ...
vinodh_eee's user avatar
0 votes
1 answer
20 views

Efficient anomaly detection in unordered market data - is it possible?

I'm a little bit stuck on how to efficiently model anomaly detection for the following problem, probably because of my lack of experience with time series modelling: I retrieve market data sorted by ...
Skyence's user avatar
0 votes
0 answers
6 views

How to Identify Equipment Churn from Laboratory Service Records Without Direct Churn Labels?

I'm analyzing a dataset encompassing 20 years of laboratory equipment service records, which includes the equipment ID, service dates, types of equipment (HOOD_TYPE), and descriptions of performed ...
tlengman's user avatar
0 votes
0 answers
15 views

Addressing prolonged high matrix profile values in anomaly detection

In an anomaly detection task, I have a data stream where each new data point is generated every 5 minutes. When a new data point arrives, I compute the matrix profile using Stumpy's stumpi function. ...
Vlad's user avatar
  • 101
0 votes
0 answers
44 views

Unsupervised Log Anomaly Detection

I am thinking about using the variational autoencoder model for anomaly detection . I have an Android Logs dataset. As the logs generated are a representative of time series type of data I thought ...
MLenthusiast's user avatar
0 votes
1 answer
57 views

Anomaly Detection in Log Data using LSTM

Problem Overview: I am currently working on a project involving anomaly detection in log data. The anomalies are defined by deviations from historical patterns. The log data has a simple structure: [...
Raj's user avatar
  • 1
0 votes
1 answer
19 views

Univariate anomaly / outlier detection

I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection. I have a dataset of around 60K data points. Each data point is part ...
EyalG's user avatar
  • 1
0 votes
2 answers
125 views

How to detect and predict sensor faults and failures (for weather stations to be specific)?

Need help. Especially those knowledgeable in weather systems/meteorology. Best approach in detecting and predicting faulty weather sensors and their failures based on their readings alone? I'm doing a ...
noob101's user avatar
0 votes
0 answers
50 views

Improving performance of anomaly detection using dataset?

I am leveraging an isolation forest model from the scikit-learn library for anomaly detection in a time series dataset where each point in the dataset is a data frame. However, I possess additional ...
SUNITA GUPTA's user avatar
0 votes
0 answers
78 views

Modelling LSTM Autoencoder for anomaly detection with multiple Time Series

I'm currently working on LSTM autoencoder for anomaly detection. My main problem is I have multiple time series - each individual time series corresponds to a different customer, detailing their sales ...
suziex's user avatar
  • 1
1 vote
0 answers
148 views

How can I apply NLP/NLU methods for anomaly detection in structured log data?

I have a dataset of logs with a specific structured format, and I'm looking for the best approach to detect anomalies within this data. I've already experimented with autoencoders and clustering ...
Coolfrog's user avatar
0 votes
0 answers
27 views

Detecting Anomalies Using LSTMs

I'm studying this article. The authors used a two-step approach to detect anomalies. First, they used an LSTM to learn the normal behavior of the data. Then, they used the dynamic error thresholds to ...
Vahid Shams's user avatar
0 votes
0 answers
18 views

Unsupervised rule extraction of categorical data

I have a dataset of network traffic with three features that I would like to extract rules from in order to apply firewall/flow control rules i.e. the permitted flows. I am able to classify a ...
user153170's user avatar
0 votes
1 answer
159 views

Constructing an LSTM autoencoder for variable-lentgth sequences

I would like to construct an LSTM autoencoder model for sequence anomaly detection where the sequences can be varying in length. I understand based on this answer that padding and masking can be used ...
interoception's user avatar
1 vote
1 answer
88 views

How to Justify Anomalies Detected by Unsupervised Anomaly Detection Models? [closed]

I'm working on an unsupervised anomaly detection project involving a large sensor dataset, where I aim to identify anomalies without the aid of labeled data. While I've implemented several ...
Jais Varghese Joseph's user avatar
0 votes
1 answer
84 views

Use computer vision to detect door blockage

I want to detect door blockage on a camera. Basically if the exit door is blocked by an object, it detects it as an anomaly. How can we do it? Is it possible to do it using OpenCV? Remember, it doesn’...
Mary's user avatar
  • 217
0 votes
1 answer
21 views

Human Classification Error Detection

I'm working on a model to detect errors on human classifications. I already have a classifier model M: X -> Y, but I need now a model M' (X,Y) -> {0,1}. Y contains a lot of classes (~5-6k). My ...
EzrielS's user avatar
  • 323
0 votes
0 answers
24 views

How to build a recommender system for recommending skewed positive class examples using e.g. one-class classification model or anomaly detection?

I want to build a recommendation system for accounts purchasing some items. The ratio of purchase events to view events is very low (less than 1% items that are viewed get bought). Right now I am ...
exAres's user avatar
  • 251
0 votes
0 answers
72 views

What is the best way to approach anomaly detection on a data set using machine learning?

I am looking to help on where to start exploring machine learning when it comes to data processing. Say I have the following csv file with hundreds of thousands of rows of data: ID Amount Overdue (...
Doug's user avatar
  • 1
0 votes
0 answers
24 views

What is the exctual purpose of a real-time log anomaly detection?

I'm sorry for asking a bit outlier topic question. Currently, I'm trying to understand log parser processing and log analysis for anomaly detection in the ML/DL class. There are so many references for ...
orde.r's user avatar
  • 113
0 votes
1 answer
150 views

Anomaly Detection: Large number of categories

Looking for some advice. I am working on an Anomaly detection problem, I am looking at parcels being transported from A-B and want to identify which parcels are considered anomalies for given routes. ...
darren's user avatar
  • 1
0 votes
1 answer
29 views

Best practice labeling grouped anomalies for object detection

I would like to train object detection model (e.g. YOLO) for images that contain anomalies. The anomalies are essentially the holes in a surface of different sizes. How do I label correctly such ...
In777's user avatar
  • 3
0 votes
0 answers
24 views

Varying feature vector lengths for learning

[I am a total beginner in machine learning algorithms] I have 10 spectrograms (lines) for phytoplankton (each composed of 288 points). Each spectrogram is associated with a phytoplankton dendity data ...
Marianne's user avatar
0 votes
0 answers
88 views

Performance metrics for outlier/anomaly detection

I am currently seeking for metrics i can use to evaluate a model for outlier/anomaly detection without ground truth. The only thing i came up with for now is to use the scores/probas returned by my ...
Raphael's user avatar
0 votes
1 answer
14 views

RCA on top of existing anomalies

When we work with timeseries data containing multiple features eg. a sensor data. we can detect anomalies using cluster, supervised and semi supervised based approaches (Eg. Isolation, Autoencoder Etc)...
duck's user avatar
  • 111
0 votes
1 answer
59 views

Unsupervised anomaly detection - dataset with multiple users

I'm confused on how to go about an issue. I'm trying to implement an unsupervised model, using a dataset that is essentially a log file. This dataset contains a variety of features, but most ...
Kurt's user avatar
  • 1
1 vote
0 answers
79 views

Threshold for auto encoder anomaly detection

I have fitted an auto encoder on 25-dimensional time series data hoping to be able to detect anomalies. training set is 100k observations, testset for threshold setting is 10k observations. all ...
Student's user avatar
  • 21
1 vote
1 answer
40 views

Best way to detect newly incoming anomalies in two timeseries?

I have two devices that both send data (let's say temperature). I need to be able to detect if one of the devices reports an unusual/anomalous reading. The case if both of the devices report a "...
Jamess11's user avatar
2 votes
1 answer
320 views

Early anomaly detection / Failure prediction on time series

My problem here is that I want to predict failures in advance with respect to their occurrence. I have sensors mounted on my machine and with a certain frequency, they send data to my database. ...
Andrea's user avatar
  • 45
1 vote
1 answer
22 views

Underfitting and perfomance metrics in unsupervised methods

My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
BilboBuggins's user avatar
-2 votes
1 answer
139 views

How to increase , precision-recall value in your Deep learning model

I am getting good accuracy metrics around 80 with precision =66, recall =37, F1 =47. How can I improve precision, and recall metrics in anomaly detection scenarios.. any suggestions?
user12's user avatar
  • 171
1 vote
1 answer
127 views

Determining threshold for KMeans anomaly detection

I'm trying to use KMeans for anomaly detection, and I know that a threshold is needed to determine the anomalies. I've seen many articles talking about how to choose K, but none talks about how to ...
Rayne's user avatar
  • 131
0 votes
0 answers
22 views

Determine unusual occurrence of words in classes

I am working on a project where I have 20+ classes/groups. Each of these groups perform certain text searches. I am looking for specific keywords example 'code' which is an anomaly. The challenge is ...
kruparulz14's user avatar
0 votes
0 answers
48 views

Video anomaly detection/ Evaluation AUC

I have trained an unsupervised anomaly detector for surveillance videos. After inference, I rescale the scores between max/min from the resulting scores array. scores = (scores - min(scores))/max(...
TecK97's user avatar
  • 1
0 votes
1 answer
40 views

Sound Anomaly Detection

What is the recommended directory structure for sound anomaly detection using Keras CNN (Unsupervised) ? After converting the sound files into spectrograms. Code examples will be highly appreciated.
Cactus's user avatar
  • 1
0 votes
1 answer
42 views

detecting abnormality in a specific feature with respect to others (unsupervised?)

I have a large dataset with a feature y which is dependent in part on features x1 and x2. All features are noisy, and y is also dependent on other parameters not captured in the dataset. I would like ...
user18236139's user avatar
4 votes
1 answer
497 views

How to detect anomalies in web log data

I have the following challenge: We have the web logs of a platform where people can download publications and we need to detect anomalies. From time to time and only by chance we observe spikes in ...
jfix's user avatar
  • 143
0 votes
0 answers
18 views

Can/should a logical multi-class classification anomaly detection system be described as "unsupervised machine learning"?

I would like to ensure that my use of terminology is accurate. My question is: what terminology should I be using in this case? The system I am building assigns classes (-1, 0, +1) to observations ...
HackJob99's user avatar
0 votes
2 answers
420 views

Time Series - Anomaly Detection

I have time-series data with alerts (every minute) that I need to find anomalies in. I am looking for a library which can do unsupervised learning of this data and detect anomalies in the data. Which ...
Jilaba Hindga's user avatar
0 votes
0 answers
27 views

Handling features dependent on a data field that may not be present in sample

I'm trying to build an anomaly detection model using Isolation Forest. I currently have 12 features, about half of them depends on the presence of a particular data field, say ...
Rayne's user avatar
  • 131
0 votes
0 answers
35 views

Why anomaly detection with IForest does not give expected results?

My goal is to develop an anomaly detection model with Isolation Forest in order to distinguish between normal and anomalous IPs by analyzing the web access logs and ...
Kosmylo's user avatar
  • 101
1 vote
2 answers
75 views

Validate Unsupervised Binary Classification

I’m working on a fully unsupervised anomaly detection problem. Since it’s completely unsupervised, I’m having hard times in defining some metrics to kind of validate the results (I run several ...
fpialcoi_o's user avatar
0 votes
0 answers
15 views

Find the most impactfuls parameters multivariate output unsupervised ML

I am currently on a proect where my df has more than 600 parameters of analog sensors (A parameters) and about 50 other parameters (F parameters). I want to find for each of these 50 parameters (F ...
Art's user avatar
  • 1
1 vote
0 answers
60 views

How to find inconsistency in data?

I wonder how I can find any inconsistencies in data. To show what I mean let's consider the following example: Imagine I have a problem of finding out why the speed (S) of the vehicle is low. I have ...
Nourless's user avatar
  • 121
1 vote
1 answer
46 views

How can I find anomalies in features based on difference between true and predicted targets?

Generally the problem is the following : there is target (efficiency of mobile stations). The goal is to find stations which underperform and highlight the reasons of that. Other than this it is ...
Nourless's user avatar
  • 121
0 votes
2 answers
92 views

Anomaly prediction/forecasting in timeseries?

What options exist in order to forecast when next observation will be an outlier in a time series? Initially, I thought to train a simple forecasting model, which turned out to decently predict the ...
M Reyes's user avatar
0 votes
0 answers
43 views

Understanding Isolation Forest predictions

I'm running sklearn's IsolationForest on a dataset containing 2 classes of data, one that I know is the anomaly (~1.5% of the entire dataset), the other is the normal dataset. I'm using this (shuffled)...
Rayne's user avatar
  • 131

1
2 3 4 5
8