Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
1answer
29 views

Anomaly Detection in Time Series: How to label the data

How to label time series so that we can train it on machine learning models to classify data point as anomaly or not? If I have time series, and anomaly occurs at time t, should I label that point 1 ...
0
votes
0answers
11 views

QuantileTransform before or after long-to-wide format transformation?

I perform the following steps: Perform quantile transform Transform it from wide to long format Train the anomaly detection model on the wide-format data Will there be some significant change if I, ...
0
votes
1answer
13 views

CNN for unsupervised anomaly detection

I'm wondering if the following strategy has been already used and could work Let's says you have a CNN which work well to classify image data, dog and cat. You only have cat and dog image as training ...
0
votes
2answers
24 views

Comparing two time series data to find deviations between them [closed]

This is a use case that I have and I am trying to automate this. Any pointers would be helpful. Use Case: When we deploy any new version of a web service, we keep monitoring it (while deploying to ...
1
vote
1answer
15 views

Is it meaningful to use word2vec for non-string inputs like time series analysis?

I am working on a project that detects anomalies in a time series. I wonder if I can use word2vec for anomaly detection for non-string inputs like exchange rates?
1
vote
0answers
9 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
0
votes
0answers
44 views

Anomaly detection using k-means clustering in Python

I'm working on an anomaly detection task in Python. Datasets regard a collection of time series coming from a sensor, so data are timestamps and the relative values. In order to find anomalies, I'm ...
0
votes
0answers
10 views

Want to understand how Local outlier works

I am trying to understand how the local outlier factor algorithm works. I have not been able to find a decent and easy explanation of the same. I came across the post: Local Outlier Factor For ...
0
votes
1answer
21 views

Anomaly detection in a database

We have a production database. The load on the database varies at different times. I want to identify anomalies; for e.g, the number of database processes responding to user queries at 9 am is 100 for ...
0
votes
0answers
16 views

Handling Multiple Classes in Categorical variables and Modeling help

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the ...
0
votes
0answers
17 views

How do we overfit a CNN AutoEncoder for anomaly detection?

I have been working on an anomaly detection problem in which I need to treat the images of "street" as an anomaly. The images of "glaciers" will be treated as not-anomaly. The autoencoder which ...
0
votes
1answer
36 views

What kind of model is this?

Can anyone help me identify what kind of Architecture is behind this Application? Is it a "simple" Classification Network? If so how are the heatmaps generated? https://www.youtube.com/watch?v=...
0
votes
0answers
30 views

How to detect anomalies (errors and exceptions) in log files?

Is this a good approach? So I'm working on a Root Cause Analysis system which should help find the cause/the root error of failed system builds (packaged in a tarball), through the analysis of log ...
0
votes
1answer
34 views

Pandas datastructure

I'm trying to analyze database performance over a period of time and detect anomalies. The database server consists of many threads that perform different actions. I run a query to determine the ...
0
votes
0answers
28 views

Using smote for data augmentation of a data set which has no dependent variable

I am trying to use the reconstruction error obtained using an auto encoder to do novelty detection. My data set is of size (4500,55)(Note: this data doesn't have any abnormalities.When an auto-encoder ...
1
vote
0answers
31 views

Can autoencoders take time series into account?

Here, I read the following: The first key to understanding is that HTM relies on data that streams over time (...) By contrast, conventional deep learning uses static data and is therefore time ...
2
votes
0answers
44 views

unsupervised outliers detection - possible solution?

I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $ The goal is to identify anomalies in transactions data of traders. For example to ...
4
votes
1answer
120 views

Isolation forest sklearn contamination param

I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm. I'm developing in Python, more in detail using sklearn. I found out a lot of examples on this, ...
0
votes
0answers
13 views

A way to identify anomalous trends amongst several trends?

I'm working on a personal project, where I'd like to identify anomalous trends. Here's the scenario: Imagine a company can sell 3 types of say, candies: X, Y, and Z. For some reason, these prices can ...
0
votes
0answers
12 views

Script Response Time Anomaly Detection

Currently, I am figuring out how to tackle this problem. I have a dataset of response times for various scripts. These scripts each have multiple steps, of which each step has multiple sub-steps. ...
-1
votes
2answers
80 views

Anomaly detection k-means in Time Series

I'm trying to use k-means to detect anomalies in the Amount column. I have the following part of my dataset: ...
1
vote
2answers
22 views

How to find vertical clusters in 1-D data

I have residuals of a multivariate time series data obtained from sensors on a server.spikes in the plots of residuals indicate abnormal server state. I want to cluster the data into vertical clusters ...
0
votes
1answer
17 views

How to identify new clusters that the training data has never seen

I have to identify the different operational states of a server. I have readings related to the different sensors of the server ( like temp sensor,fan speed sensor,job load sensor etc).The data I have ...
1
vote
1answer
32 views

Get how similarity between the training data and the income data?

I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections I want something like a semi-supervised model. I want to predict ...
2
votes
4answers
81 views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
2
votes
5answers
153 views

Methods to detect this kind of outliers

Background I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why ...
0
votes
1answer
38 views

Median absolute deviation vs standard deviation

Can Gaussian density distributions be modified using median and median absolute deviation as opposed to mean and standard deviation (since the former are more robust)?
2
votes
0answers
85 views

Out of stock / Spike in demand prediction

The goal is to predict out-of-stock situations, either quantitatively (the gap) or qualitatively (out-of-stock likely to happen in next few weeks). Background: We have existing demand planning ...
0
votes
0answers
33 views

Image anomaly detection

I am using a conv net to classify 20 different patterns on an image. My train/test set are images where I know the class. With this, I a good enough model for my preference. It's an unbalance dataset, ...
0
votes
1answer
24 views

Clustering unbalanced dataset

The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did ...
0
votes
0answers
19 views

Methodologies and Trends for Sequence Anomaly Detection

I recently started to approach the issue of detecting deviating behavior from rule-based sequences. Basically the task is to spot any difference from "normal running" of processes that are defined as ...
1
vote
2answers
68 views

Outlier detection for Disk Space Usage

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset): ...
0
votes
2answers
40 views

What kind of learning in this training situation when anomaly detection? Supervised learning,semi-supervised learning or unsupervised learning?

I am doing anomaly detection recently, one of the methods is using AEs model to learn the pattern of normal samples. Determine it as an abnormal sample if it doesn’t match the pattern of normal ...
0
votes
0answers
36 views

Anomaly detection in time series data from multiple sensors

I've build a classification model based on 15 features coming in real time from 15 different sensors. The window time is 60 seconds (which means that the classification model needs 60 records from ...
0
votes
0answers
26 views

Which unsupervised learning algorithm can be used for peaks detection?

So, I have a dataset which has around 1388 unique products and I have to do unsupervised learning on them in order to find anomalies (high/low peaks). The data below just represents one product. The <...
0
votes
2answers
63 views

A Proper Outlier Detection For the Attached Figure

I am wondering how I could detect the red points as outliers using an algorithm (best method for this scenario) not through visualization since it is clear that they are outliers in the figure. ...
1
vote
1answer
75 views

Are Denoising Autoencoders for anomaly detection on structured data?

Can denoising autoencoders be used for anomaly detection on structured data? I know I can use denoising autoencoders for anomaly detection on images, but I don't know if they can do it for structured ...
1
vote
1answer
33 views

How to construct confidence bound for Time Series Prediction?

I have some time series data and am using some Deep Learning techniques to get its prediction. Now, I would like to construct confidence bounds for it. I calculated the residuals, their mean and ...
1
vote
1answer
28 views

Anomaly Detection System

I need a sanity check. I want to create an anomaly detection system. The logic which I am planning to use is the following: Find anomalies in the past using Seasonal Hybrid Extreme Studentized ...
2
votes
0answers
73 views

Sequence classification using oneClass SVM

In the code below, I'm using a sequence to sequence approach as a prediction model for anomaly detection. The data set I'm working with is ADFA-LD. The training phase is done using only normal ...
1
vote
1answer
130 views

Isolation Forest Score Function Theory

I am currently reading this paper on isolation forests. In the section about the score function, they mention the following. For context, $h(x)$ is definded as the path length of a data point ...
0
votes
0answers
176 views

method and dataset for credit card fraud detection

I am trying to create a machine learning model to detect credit card fraud (In our definition, fraud means chargeback). I am kinda stuck now with the dataset that I have. I don't have information on ...
0
votes
0answers
13 views

Why is my LOF algorithm producing the opposite result it should?

What could cause the local outlier factor (LOF) to output below 1.0 for outliers and above 1.0 for inliers? I have my code sort of working just by inverting the output, but I can't figure out what's ...
0
votes
1answer
789 views

Why use Variational Autoencoders VAE instead of Autoencoders AE in Anomaly Detection?

I have read many papers that recommend using Variational Autoencoders over Autoencoders since they have a more probabilistic approach and the ability to use KL divergence on the latent dimension. But ...
1
vote
0answers
36 views

Ensemble technique for combining predictions from classification and regression algorithms

Given an anomaly detection problem - A, I have divided the problem into two independent subtasks -A1 and ...
2
votes
1answer
143 views

Unsupervised learning for anomaly detection

I've started working on an anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor making machines. ...
5
votes
1answer
134 views

How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of ...
0
votes
1answer
72 views

Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
0
votes
2answers
130 views

Variable Importance in unsupervised anomaly detection algorithms

I am working on an anomaly detection problem to detect fraud in insurance claims. I have used the PyOD package and used algorithms like ABOD, CBLOF, Isolation Forest, and AutoEncoder. I couldn't find ...
0
votes
0answers
28 views

Detecting anomalies in numeric measurements

I'm working with a dataset of 162k experimental protein-peptide affinity measurements. If we ignore mostly irrelevant metadata, we are left with the following fields: protein sequence – each ...