Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
1
vote
1answer
37 views

Anomaly detection - relation between thresholds and anomalies

I'm developing an anomaly detection program in Python. Main idea is to create a new LSTM model every day, training it with the previous 7 days and predict the next day. Then, using thresholds, find ...
2
votes
1answer
26 views

ML Approach for Graph Anomaly Detection

Very new to ML. I am trying to create an anomaly detector. I have thousands of graphs like the one I have attached. I am interested in the pink line. If the pink line's behavior changes drastically ...
1
vote
1answer
7 views

Parameter Adjustment based only on tagged predictions

not sure that this is the best place to post this but if not, please let me know if there is a better stack community. I have an anomaly detection method which has some parameters. I have some data ...
0
votes
0answers
11 views

Testing if a sample fits into an existing cluster

I have a sample of data I'd like to create a model from, which would create N clusters. After the fitting to clusters, I'd like to test various samples against the existing clusters, seeing if the ...
0
votes
0answers
16 views

Can I use/modify an Autoencoder to handle missing data?

I am about to implement an Autoencoder to detect anomalies. Therefore, e.g., in my test set, there is a situation where the data stream broke for some days. This results in a lack of data and should ...
0
votes
1answer
25 views

Sampling labeled data for anomaly detection

I'm currently working on a project that requires the use of unsupervised anomaly detection, but I'm unable to find a relavent data set, so I'm considering the following option: Assuming I have a data ...
1
vote
1answer
38 views

Anomaly detection thresholds issue

I'm working on an anomaly detection development in Python. More in details, I need to analysed timeseries in order to check if anomalies are present. An anomalous value is typically a peak, so a ...
2
votes
0answers
14 views

Temporal outlier Analysis on sensor data

I am working to find anomaly/outliers in sensor data using unsupervised machine learning (without training dataset). I have around 20000 samples taken per minute of various sensors. I just need to ...
0
votes
0answers
7 views

Perform unsupervised anomaly identification with causation in Python?

I have a time series data, which contains information from various sensors measured at every 20 minutes interval. I would like to use information from all these sensors as features to a Deep Learning/...
0
votes
1answer
47 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
0
votes
0answers
11 views

VAE latent space dimension with Weibull PDFs as input data

I am currently using a Variational Autoencoder to reconstruct PDFs of Weibull distributions with varying parameters (sampled uniformly from a given parameter span). The PDFs are generated by sampling ...
0
votes
0answers
10 views

LSTM based anomaly detection scheme too closely tracking long spans of anomalous points

I've built a time series anomaly detection process that accurately predicts the value at the next interval. However, when there are dozens of anomalous events in a row, the model starts to "catch up" ...
1
vote
1answer
19 views

Functions in scikit that detect outliers automatically?

I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?
0
votes
2answers
280 views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
0
votes
0answers
29 views

dataset shift, covariate shift or sample selection bias in small subsets?

I try to show from which training data size on which machine learning method (CNN, SVM etc.) achieves better performance. For this I would like to use subsets of different sizes from the datasets of ...
0
votes
2answers
43 views

Is it safe to use labels created from unsupervised model to train a supervised model using the same data?

I have a dataset where I have to detect anomalies. Now, I use a subset of the data(let's call that subset A) and apply the DBSCAN algorithm to detect anomalies on set A.Once the anomalies are detected,...
4
votes
0answers
27 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
1
vote
2answers
53 views

anomaly detection - conceptual help on machine learning

I am working on Anomaly detection model problem for a finance data set - set of card transactions. My team member suggested an idea that " First train the model with normal instances of data and to ...
0
votes
1answer
28 views

How to implement Classification and Anomaly detection (C++)

I am creating a system using C++(DX11) and i'm reading raw data into my program, i want to classify what the 3D data-set i'm reading in is and detect any anomalies it may have when compared to a ...
0
votes
0answers
18 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
0answers
8 views

How to solve a classification problem with multiple time series?

I am trying to build a model for credit default prediction. I've got a dataset of over 20,000 customers and the features are their payments over the last ≤24 months. The dataset looks like this: <...
0
votes
3answers
46 views

Dealing with informative missingness

How can I deal with a time series that contains missing data which means something? So the value that is missing is not wrong. It's missing on purpose and imputing those missing values would mean a ...
0
votes
1answer
31 views

Anomaly detection for time series data with only positive samples?

I'm having a time series ECG dataset. I want to do anomaly detection (anything different from normal ECG should be abnormal). The point is I'm having only positive samples with very few negative ...
0
votes
2answers
92 views

Anomaly Detection in Time Series: How to label the data

How to label time series so that we can train it on machine learning models to classify data point as anomaly or not? If I have time series, and anomaly occurs at time t, should I label that point 1 ...
0
votes
0answers
12 views

QuantileTransform before or after long-to-wide format transformation?

I perform the following steps: Perform quantile transform Transform it from wide to long format Train the anomaly detection model on the wide-format data Will there be some significant change if I, ...
0
votes
1answer
43 views

CNN for unsupervised anomaly detection

I'm wondering if the following strategy has been already used and could work Let's says you have a CNN which work well to classify image data, dog and cat. You only have cat and dog image as training ...
1
vote
2answers
40 views

Comparing two time series data to find deviations between them [closed]

This is a use case that I have and I am trying to automate this. Any pointers would be helpful. Use Case: When we deploy any new version of a web service, we keep monitoring it (while deploying to ...
1
vote
2answers
93 views

Is it meaningful to use word2vec for non-string inputs like time series analysis?

I am working on a project that detects anomalies in a time series. I wonder if I can use word2vec for anomaly detection for non-string inputs like exchange rates?
1
vote
0answers
20 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
0
votes
0answers
738 views

Anomaly detection using k-means clustering in Python

I'm working on an anomaly detection task in Python. Datasets regard a collection of time series coming from a sensor, so data are timestamps and the relative values. In order to find anomalies, I'm ...
0
votes
1answer
31 views

Anomaly detection in a database

We have a production database. The load on the database varies at different times. I want to identify anomalies; for e.g, the number of database processes responding to user queries at 9 am is 100 for ...
0
votes
0answers
20 views

Handling Multiple Classes in Categorical variables and Modeling help

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the ...
0
votes
0answers
99 views

How do we overfit a CNN AutoEncoder for anomaly detection?

I have been working on an anomaly detection problem in which I need to treat the images of "street" as an anomaly. The images of "glaciers" will be treated as not-anomaly. The autoencoder which ...
0
votes
1answer
36 views

What kind of model is this?

Can anyone help me identify what kind of Architecture is behind this Application? Is it a "simple" Classification Network? If so how are the heatmaps generated? https://www.youtube.com/watch?v=...
0
votes
0answers
49 views

How to detect anomalies (errors and exceptions) in log files?

Is this a good approach? So I'm working on a Root Cause Analysis system which should help find the cause/the root error of failed system builds (packaged in a tarball), through the analysis of log ...
0
votes
1answer
36 views

Pandas datastructure

I'm trying to analyze database performance over a period of time and detect anomalies. The database server consists of many threads that perform different actions. I run a query to determine the ...
0
votes
0answers
42 views

Using smote for data augmentation of a data set which has no dependent variable

I am trying to use the reconstruction error obtained using an auto encoder to do novelty detection. My data set is of size (4500,55)(Note: this data doesn't have any abnormalities.When an auto-encoder ...
2
votes
0answers
70 views

Can autoencoders take time series into account?

Here, I read the following: The first key to understanding is that HTM relies on data that streams over time (...) By contrast, conventional deep learning uses static data and is therefore time ...
2
votes
0answers
51 views

unsupervised outliers detection - possible solution?

I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $ The goal is to identify anomalies in transactions data of traders. For example to ...
5
votes
1answer
542 views

Isolation forest sklearn contamination param

I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm. I'm developing in Python, more in detail using sklearn. I found out a lot of examples on this, ...
0
votes
0answers
15 views

A way to identify anomalous trends amongst several trends?

I'm working on a personal project, where I'd like to identify anomalous trends. Here's the scenario: Imagine a company can sell 3 types of say, candies: X, Y, and Z. For some reason, these prices can ...
0
votes
0answers
15 views

Script Response Time Anomaly Detection

Currently, I am figuring out how to tackle this problem. I have a dataset of response times for various scripts. These scripts each have multiple steps, of which each step has multiple sub-steps. ...
-1
votes
2answers
237 views

Anomaly detection k-means in Time Series

I'm trying to use k-means to detect anomalies in the Amount column. I have the following part of my dataset: ...
1
vote
2answers
30 views

How to find vertical clusters in 1-D data

I have residuals of a multivariate time series data obtained from sensors on a server.spikes in the plots of residuals indicate abnormal server state. I want to cluster the data into vertical clusters ...
0
votes
1answer
22 views

How to identify new clusters that the training data has never seen

I have to identify the different operational states of a server. I have readings related to the different sensors of the server ( like temp sensor,fan speed sensor,job load sensor etc).The data I have ...
1
vote
1answer
37 views

Get how similarity between the training data and the income data?

I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections I want something like a semi-supervised model. I want to predict ...
2
votes
5answers
440 views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
2
votes
5answers
186 views

Methods to detect this kind of outliers

Background I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why ...
0
votes
1answer
41 views

Median absolute deviation vs standard deviation

Can Gaussian density distributions be modified using median and median absolute deviation as opposed to mean and standard deviation (since the former are more robust)?
2
votes
0answers
92 views

Out of stock / Spike in demand prediction

The goal is to predict out-of-stock situations, either quantitatively (the gap) or qualitatively (out-of-stock likely to happen in next few weeks). Background: We have existing demand planning ...