Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

1
vote
0answers
5 views

Sequence classification using oneClass SVM

In the code below, I'm using a sequence to sequence approach as a prediction model for anomaly detection, The data set I'm working with is ADFA-LD. The training phase is done using only normal ...
1
vote
1answer
17 views

Isolation Forest Score Function Theory

I am currently reading this paper on isolation forests. In the section about the score function, they mention the following. For context, $h(x)$ is definded as the path length of a data point ...
0
votes
0answers
32 views

method and dataset for credit card fraud detection

I am trying to create a machine learning model to detect credit card fraud (In our definition, fraud means chargeback). I am kinda stuck now with the dataset that I have. I don't have information on ...
0
votes
0answers
11 views

Why is my LOF algorithm producing the opposite result it should?

What could cause the local outlier factor (LOF) to output below 1.0 for outliers and above 1.0 for inliers? I have my code sort of working just by inverting the output, but I can't figure out what's ...
0
votes
1answer
259 views

Why use Variational Autoencoders VAE insted of Autoencoders AE in Anomaly Detection?

I have read many papers that recommends using Variational Autoencoders over Autoencoders since they have a more probabilistic approach and the ability to use KL divergence on the latent dimension. But ...
1
vote
0answers
30 views

Ensemble technique for combining predictions from classification and regression algorithms

Given an anomaly detection problem - A, I have divided the problem into two independent subtasks -A1 and ...
1
vote
1answer
46 views

Unsupervised learning for anomaly detection

I've started working on an anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor making machines. ...
5
votes
1answer
87 views

How to compare two unsupervised anomaly detection algorithms on the same data-set?

I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of ...
0
votes
1answer
37 views

Algorithm suggestion for anomaly detection in multivariate time series data

I have time series data containing user actions at certain time intervals eg ...
0
votes
2answers
51 views

Variable Importance in unsupervised anomaly detection algorithms

I am working on an anomaly detection problem to detect fraud in insurance claims. I have used the PyOD package and used algorithms like ABOD, CBLOF, Isolation Forest, and AutoEncoder. I couldn't find ...
0
votes
0answers
27 views

Detecting anomalies in numeric measurements

I'm working with a dataset of 162k experimental protein-peptide affinity measurements. If we ignore mostly irrelevant metadata, we are left with the following fields: protein sequence – each ...
2
votes
3answers
70 views

Unsupervised Anomaly Detection on system metrics like memory, cpu, io, net, etc

In all the examples that I can see online, people have used a labelled dataset. I however am stuck trying to construct a model to perform anomaly detection on unlabelled dataset (unsupervised anomaly ...
0
votes
1answer
26 views

How to represent unbounded number of items in a DNN

I have an Order defined as { "order_id": 123, "total_cost": 5000, "items": [ { "name": "Buffalo Wings", "cost": 1500 }, { "name": "Fries", "cost": 500 ...
1
vote
0answers
14 views

Anomaly detection in structured textual data

Pls refer screenshot for sample data. As can be seen most of the fields in data are textual and highly correlated but each row has unique values and hence won't be right to call it categorical. I ...
0
votes
1answer
78 views

Anomaly detection on multidimensional time series

I have relatively little knowledge of unsupervised machine learning. I'm working on a project that aims to find anomalies in a set of n data, measured every ...
0
votes
1answer
18 views

which outlier detection technique?

I'm new to data science. I have a query on anomaly detection techniques. There are several anomaly detection techniques Like statistical, density based, depth based , clustering etc.. Given a ...
1
vote
0answers
19 views

Feature encoding for multiple JSON objects

I have a dataset, where a particular feature is a collection of many JSON objects for a single feature. ...
1
vote
0answers
45 views

Anomaly detection on text data using one Class SVM

I am working on an Anomaly detection problem that is first of its kind for me. I am able to do data cleansing and train the model however I have no idea on predicting results, as it is new for me and ...
1
vote
1answer
61 views

DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?

I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection. Clustering is done ...
0
votes
0answers
8 views

Long run time for grid search SARIMA

I am running a grid search for identifying the right set of params for Seasonal ARIMA, for over a 1300 training set and range for all the params being 0,1 and 2. But this process is taking over ...
0
votes
0answers
42 views

Anomaly detection in text - how to utilize ngram frequency of words for the detection of an anomalous document?

I am calculating ngram frequency on a text, and for each word I output its conditional probability: how frequent it is given thew last n-1 words, or, if you will, p(ngram)/p([n-1]gram). Using this I ...
0
votes
0answers
74 views

Comparison between approaches for timeseries anomaly detection

After various days of research, I could take a global picture of the existing methods to perform anomaly detection on time series, namely: Forecasting with Deep Learning. Eg. RADM or LSTM model ...
0
votes
0answers
10 views

Which methods exist to find correlations between multiple univariate timeseries anomaly detection output?

In this short article from Anodot, they explain the (dis)advantages of directly applying a multivariate anomaly detection model on raw timeseries data. Instead, they look for anomalies on each ...
0
votes
0answers
36 views

Forecasting vs non-forecasting predition for time series anomaly detection

I have got the objective of implementing a uni/multivariate online anomaly detection system. After multiple days of research, I could collect many ways to achieve this (Eg. moving average solutions ...
2
votes
0answers
43 views

Anomaly Detection: Model Creation & Implementation

I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information. ...
1
vote
0answers
16 views

Detecting timestamp and feature where anomaly occurs with autoencoder

Is it possible according to you to use the latent space of a deep autoencoder to detect the index of the sample and the column/s (features) where an anomaly occurs or am I losing information with ...
1
vote
1answer
38 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
-1
votes
1answer
50 views

Finding outliers from multiple files [closed]

I am dealing with a very strange problem. I have a lot of files. I need to show which files are similar and which one has exception/outliers using its data. I can show with unsupervised learning ...
1
vote
1answer
39 views

Do anomalous input features to autoencoder result in high errors on the corresponding output features?

An autoencoder is trained by replicating each training instance to both input and output. However, when predicting for anomaly detection, will the output error be local to the same output feature(s) ...
1
vote
0answers
15 views

Practical examples/tutorials of using One-Class Support Vector Machines

I am a newbie in machine learning, and hope to solve an anomaly detection task using One-Class Support Vector Machines (OCSVM). ...
0
votes
1answer
85 views

How can I detect anomalies/outliers in my online streaming data on a real-time basis?

Say, I've a huge set of data(infinite in size) consisting of alternating sine wave and step pulses one after the other. What I want from my model is to parse the data sequence wise or point wise and ...
0
votes
1answer
60 views

Sequences of time series data with only 1 output classification

So I'm facing a problem where I have a sequence (30h of data with 10sec intervals) and which is labeled to a class (3 different classes) for the whole sequence. I'm used to work with time series who ...
0
votes
1answer
35 views

Explaination of the anomalies detected

I have a dataset with a mixture of categorical and numeric variables. I have converted categorical variables into one hot encoding. I have used Isolation forest algorithm to extract 5% of the ...
1
vote
1answer
198 views

What are the techniques for anomaly detection of Unsupervised learning problem

I have sufficient and properly formatted data in millions without labels. I have to find out the anomalies. Heard Isolation forest, Mahalanobis distance about identifying anomalies in unsupervised ...
0
votes
2answers
50 views

Skewed two class data set

Is there any theory on the influence of skew in the data set on the performance of binary classifiers? At work, we are doing abuse detection, the negative population is regular logins, and the ...
1
vote
1answer
1k views

Validation loss is lower than the training loss

I am using autoencoder for anomaly detection in warranty data. Architecture 1: The plot shows the training vs validation loss based on Architecture 1. As we see in the plot, validation loss is ...
1
vote
1answer
36 views

Is train/test-Split in unsupervised learning of neural network necessary?

I am using autoencoder for anomaly detection in warranty data. It is unsupervised. I calculate the reconstruction error by the model and the records with high reconstruction error value is considered ...
-1
votes
1answer
52 views

How to use K-Means to detect users anomaly in Access Control

I'm currently working on access control project, Smart Lock to be more spesific. Like the other smart lock system, the system required user's authentication to open the door. I'm using RFID as ...
1
vote
2answers
78 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
2
votes
4answers
59 views

i"m confused that how to apply k means in my dataset

I have to detect anomalies from my data-set. The anomaly is about in which area and in which time of network usage(total_activity in my data) drastically improved. Help me to know how to apply k-means ...
1
vote
2answers
36 views

What is a good method for detection of rare occurencies of speech in noisy audio data?

I have some audio recordings (with relatively static but noisy background, e.g., wind in an open area) with small number of short occurrences of speech (~1% of the total audio duration). What would ...
1
vote
0answers
14 views

What do you call an anomaly/signature detection algorithm in data science

If I have an algorithm for detecting a set of data points that indicate with a high level of certainty that some event has occured OR that behaviors outside of a set model are occuring. What would ...
0
votes
1answer
34 views

is there a way to check if i got a “good price” on something?

I'm looking at some data. Actually, the Boston Housing dataset is probably a good proxy for it: https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html I'm wondering if there's a way to ...
2
votes
1answer
37 views

Given data that is labeled as outliers, how can I classify data as outliers?

I have a dataset that is a mixture of sparse binary features and quantitative features. I only have definite outliers labeled. How should I approach trying to classify unlabeled data? I considered ...
2
votes
2answers
7k views

Anomaly detection on time series

I've just started to working on an anomaly detection development in Python. My data sets regard a collection of timeseries. More in details, data are coming from some sensors/meters which record and ...
1
vote
2answers
239 views

Cross validation for anomaly detection using autoencoder

I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't ...
1
vote
2answers
103 views

How can I find anomalies in each row of data?

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility. ...
0
votes
0answers
26 views

How to define an anomaly detection problem?

I defined a supervised machine learning tasks as: Given: $m$ training examples with $n-1$ features: $E\in\mathcal{R}^{m\times (n-1)}$ and labels: $L \in \mathcal{R}^m$, a model $f$ of $d$ ...
1
vote
1answer
147 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of ...
1
vote
1answer
172 views

Using DONUT algorithm with keras

I am trying to get this repo of Xu's DONUT algorithm running, however I am getting an error I do not quite understand. The readme says I should load raw_data as follows: ...