Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

0
votes
0answers
10 views

Script Response Time Anomaly Detection

Currently, I am figuring out how to tackle this problem. I have a dataset of response times for various scripts. These scripts each have multiple steps, of which each step has multiple sub-steps. ...
-1
votes
2answers
29 views

Anomaly detection k-means in Time Series

I'm trying to use k-means to detect anomalies in the Amount column. I have the following part of my dataset: ...
1
vote
2answers
16 views

How to find vertical clusters in 1-D data

I have residuals of a multivariate time series data obtained from sensors on a server.spikes in the plots of residuals indicate abnormal server state. I want to cluster the data into vertical clusters ...
0
votes
1answer
14 views

How to identify new clusters that the training data has never seen

I have to identify the different operational states of a server. I have readings related to the different sensors of the server ( like temp sensor,fan speed sensor,job load sensor etc).The data I have ...
1
vote
1answer
29 views

Get how similarity between the training data and the income data?

I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections I want something like a semi-supervised model. I want to predict ...
2
votes
4answers
41 views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
0
votes
1answer
28 views

Median absolute deviation vs standard deviation

Can Gaussian density distributions be modified using median and median absolute deviation as opposed to mean and standard deviation (since the former are more robust)?
2
votes
0answers
81 views

Out of stock / Spike in demand prediction

The goal is to predict out-of-stock situations, either quantitatively (the gap) or qualitatively (out-of-stock likely to happen in next few weeks). Background: We have existing demand planning ...
0
votes
0answers
22 views

Image anomaly detection

I am using a conv net to classify 20 different patterns on an image. My train/test set are images where I know the class. With this, I a good enough model for my preference. It's an unbalance dataset, ...
0
votes
1answer
23 views

Clustering unbalanced dataset

The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did ...
0
votes
0answers
11 views

Methodologies and Trends for Sequence Anomaly Detection

I recently started to approach the issue of detecting deviating behavior from rule-based sequences. Basically the task is to spot any difference from "normal running" of processes that are defined as ...
1
vote
2answers
57 views

Outlier detection for Disk Space Usage

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset): ...
0
votes
2answers
26 views

What kind of learning in this training situation when anomaly detection? Supervised learning,semi-supervised learning or unsupervised learning?

I am doing anomaly detection recently, one of the methods is using AEs model to learn the pattern of normal samples. Determine it as an abnormal sample if it doesn’t match the pattern of normal ...
0
votes
0answers
22 views

Anomaly detection in time series data from multiple sensors

I've build a classification model based on 15 features coming in real time from 15 different sensors. The window time is 60 seconds (which means that the classification model needs 60 records from ...
0
votes
0answers
23 views

Which unsupervised learning algorithm can be used for peaks detection?

So, I have a dataset which has around 1388 unique products and I have to do unsupervised learning on them in order to find anomalies (high/low peaks). The data below just represents one product. The <...
0
votes
2answers
57 views

A Proper Outlier Detection For the Attached Figure

I am wondering how I could detect the red points as outliers using an algorithm (best method for this scenario) not through visualization since it is clear that they are outliers in the figure. ...
1
vote
1answer
34 views

Are Denoising Autoencoders for anomaly detection on structured data?

Can denoising autoencoders be used for anomaly detection on structured data? I know I can use denoising autoencoders for anomaly detection on images, but I don't know if they can do it for structured ...
1
vote
1answer
30 views

How to construct confidence bound for Time Series Prediction?

I have some time series data and am using some Deep Learning techniques to get its prediction. Now, I would like to construct confidence bounds for it. I calculated the residuals, their mean and ...
0
votes
1answer
24 views

Anomaly Detection System

I need a sanity check. I want to create an anomaly detection system. The logic which I am planning to use is the following: Find anomalies in the past using Seasonal Hybrid Extreme Studentized ...
1
vote
0answers
35 views

Sequence classification using oneClass SVM

In the code below, I'm using a sequence to sequence approach as a prediction model for anomaly detection, The data set I'm working with is ADFA-LD. The training phase is done using only normal ...
1
vote
1answer
65 views

Isolation Forest Score Function Theory

I am currently reading this paper on isolation forests. In the section about the score function, they mention the following. For context, $h(x)$ is definded as the path length of a data point ...
0
votes
0answers
97 views

method and dataset for credit card fraud detection

I am trying to create a machine learning model to detect credit card fraud (In our definition, fraud means chargeback). I am kinda stuck now with the dataset that I have. I don't have information on ...
0
votes
0answers
11 views

Why is my LOF algorithm producing the opposite result it should?

What could cause the local outlier factor (LOF) to output below 1.0 for outliers and above 1.0 for inliers? I have my code sort of working just by inverting the output, but I can't figure out what's ...
0
votes
1answer
550 views

Why use Variational Autoencoders VAE instead of Autoencoders AE in Anomaly Detection?

I have read many papers that recommend using Variational Autoencoders over Autoencoders since they have a more probabilistic approach and the ability to use KL divergence on the latent dimension. But ...
1
vote
0answers
32 views

Ensemble technique for combining predictions from classification and regression algorithms

Given an anomaly detection problem - A, I have divided the problem into two independent subtasks -A1 and ...
2
votes
1answer
90 views

Unsupervised learning for anomaly detection

I've started working on an anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor making machines. ...
5
votes
1answer
106 views

How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of ...
0
votes
1answer
57 views

Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
0
votes
2answers
87 views

Variable Importance in unsupervised anomaly detection algorithms

I am working on an anomaly detection problem to detect fraud in insurance claims. I have used the PyOD package and used algorithms like ABOD, CBLOF, Isolation Forest, and AutoEncoder. I couldn't find ...
0
votes
0answers
28 views

Detecting anomalies in numeric measurements

I'm working with a dataset of 162k experimental protein-peptide affinity measurements. If we ignore mostly irrelevant metadata, we are left with the following fields: protein sequence – each ...
2
votes
3answers
113 views

Unsupervised Anomaly Detection on system metrics like memory, cpu, io, net, etc

In all the examples that I can see online, people have used a labelled dataset. I however am stuck trying to construct a model to perform anomaly detection on unlabelled dataset (unsupervised anomaly ...
0
votes
1answer
27 views

How to represent unbounded number of items in a DNN

I have an Order defined as { "order_id": 123, "total_cost": 5000, "items": [ { "name": "Buffalo Wings", "cost": 1500 }, { "name": "Fries", "cost": 500 ...
1
vote
0answers
23 views

Anomaly detection in structured textual data

Pls refer screenshot for sample data. As can be seen most of the fields in data are textual and highly correlated but each row has unique values and hence won't be right to call it categorical. I ...
0
votes
1answer
218 views

Anomaly detection on multidimensional time series

I have relatively little knowledge of unsupervised machine learning. I'm working on a project that aims to find anomalies in a set of n data, measured every ...
0
votes
1answer
58 views

which outlier detection technique?

I'm new to data science. I have a question on anomaly detection techniques. There are several anomaly detection techniques such as statistical, density based, depth based, clustering, etc.. Given a ...
1
vote
0answers
32 views

Feature encoding for multiple JSON objects

I have a dataset, where a particular feature is a collection of many JSON objects for a single feature. ...
1
vote
0answers
71 views

Anomaly detection on text data using one Class SVM

I am working on an Anomaly detection problem that is first of its kind for me. I am able to do data cleansing and train the model however I have no idea on predicting results, as it is new for me and ...
1
vote
1answer
126 views

DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?

I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection. Clustering is done ...
0
votes
0answers
16 views

Long run time for grid search SARIMA

I am running a grid search for identifying the right set of params for Seasonal ARIMA, for over a 1300 training set and range for all the params being 0,1 and 2. But this process is taking over ...
0
votes
0answers
85 views

Anomaly detection in text - how to utilize ngram frequency of words for the detection of an anomalous document?

I am calculating ngram frequency on a text, and for each word I output its conditional probability: how frequent it is given thew last n-1 words, or, if you will, p(ngram)/p([n-1]gram). Using this I ...
0
votes
0answers
182 views

Comparison between approaches for timeseries anomaly detection

After various days of research, I could take a global picture of the existing methods to perform anomaly detection on time series, namely: Forecasting with Deep Learning. Eg. RADM or LSTM model ...
0
votes
0answers
12 views

Which methods exist to find correlations between multiple univariate timeseries anomaly detection output?

In this short article from Anodot, they explain the (dis)advantages of directly applying a multivariate anomaly detection model on raw timeseries data. Instead, they look for anomalies on each ...
0
votes
0answers
91 views

Forecasting vs non-forecasting predition for time series anomaly detection

I have got the objective of implementing a uni/multivariate online anomaly detection system. After multiple days of research, I could collect many ways to achieve this (Eg. moving average solutions ...
2
votes
0answers
50 views

Anomaly Detection: Model Creation & Implementation

I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information. ...
1
vote
0answers
21 views

Detecting timestamp and feature where anomaly occurs with autoencoder

Is it possible according to you to use the latent space of a deep autoencoder to detect the index of the sample and the column/s (features) where an anomaly occurs or am I losing information with ...
1
vote
1answer
42 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
-1
votes
1answer
51 views

Finding outliers from multiple files [closed]

I am dealing with a very strange problem. I have a lot of files. I need to show which files are similar and which one has exception/outliers using its data. I can show with unsupervised learning ...
1
vote
1answer
48 views

Do anomalous input features to autoencoder result in high errors on the corresponding output features?

An autoencoder is trained by replicating each training instance to both input and output. However, when predicting for anomaly detection, will the output error be local to the same output feature(s) ...
1
vote
0answers
17 views

Practical examples/tutorials of using One-Class Support Vector Machines

I am a newbie in machine learning, and hope to solve an anomaly detection task using One-Class Support Vector Machines (OCSVM). ...
0
votes
1answer
104 views

How can I detect anomalies/outliers in my online streaming data on a real-time basis?

Say, I've a huge set of data(infinite in size) consisting of alternating sine wave and step pulses one after the other. What I want from my model is to parse the data sequence wise or point wise and ...