Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [anomaly-detection]

The tag has no usage guidance.

1
vote
0answers
10 views

DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?

I'm trying to understand the paper : https://netman.aiops.org/wp-content/uploads/2018/05/PID5338621.pdf about Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection. Clustering is done ...
0
votes
0answers
6 views

Long run time for grid search SARIMA

I am running a grid search for identifying the right set of params for Seasonal ARIMA, for over a 1300 training set and range for all the params being 0,1 and 2. But this process is taking over ...
0
votes
0answers
15 views

Anomaly detection in text - how to utilize ngram frequency of words for the detection of an anomalous document?

I am calculating ngram frequency on a text, and for each word I output its conditional probability: how frequent it is given thew last n-1 words, or, if you will, p(ngram)/p([n-1]gram). Using this I ...
0
votes
0answers
20 views

Comparison between approaches for timeseries anomaly detection

After various days of research, I could take a global picture of the existing methods to perform anomaly detection on time series, namely: Forecasting with Deep Learning. Eg. RADM or LSTM model ...
0
votes
0answers
7 views

Which methods exist to find correlations between multiple univariate timeseries anomaly detection output?

In this short article from Anodot, they explain the (dis)advantages of directly applying a multivariate anomaly detection model on raw timeseries data. Instead, they look for anomalies on each ...
0
votes
0answers
15 views

Forecasting vs non-forecasting predition for time series anomaly detection

I have got the objective of implementing a uni/multivariate online anomaly detection system. After multiple days of research, I could collect many ways to achieve this (Eg. moving average solutions ...
2
votes
0answers
28 views

Anomaly Detection: Model Creation & Implementation

I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information. ...
1
vote
0answers
11 views

Detecting timestamp and feature where anomaly occurs with autoencoder

Is it possible according to you to use the latent space of a deep autoencoder to detect the index of the sample and the column/s (features) where an anomaly occurs or am I losing information with ...
0
votes
0answers
157 views

Outliner detection with LSTM autoencoder

I am learning about autoencoders for outlier detection. I have searched enough and internet suggest to use LSTM autoencoders for outlier detection from multivariant time series data. I have watched ...
1
vote
1answer
32 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
-1
votes
1answer
50 views

Finding outliers from multiple files [closed]

I am dealing with a very strange problem. I have a lot of files. I need to show which files are similar and which one has exception/outliers using its data. I can show with unsupervised learning ...
1
vote
1answer
26 views

Do anomalous input features to autoencoder result in high errors on the corresponding output features?

An autoencoder is trained by replicating each training instance to both input and output. However, when predicting for anomaly detection, will the output error be local to the same output feature(s) ...
1
vote
0answers
15 views

Practical examples/tutorials of using One-Class Support Vector Machines

I am a newbie in machine learning, and hope to solve an anomaly detection task using One-Class Support Vector Machines (OCSVM). ...
0
votes
1answer
54 views

How can I detect anomalies/outliers in my online streaming data on a real-time basis?

Say, I've a huge set of data(infinite in size) consisting of alternating sine wave and step pulses one after the other. What I want from my model is to parse the data sequence wise or point wise and ...
0
votes
1answer
40 views

Sequences of time series data with only 1 output classification

So I'm facing a problem where I have a sequence (30h of data with 10sec intervals) and which is labeled to a class (3 different classes) for the whole sequence. I'm used to work with time series who ...
0
votes
1answer
29 views

Explaination of the anomalies detected

I have a dataset with a mixture of categorical and numeric variables. I have converted categorical variables into one hot encoding. I have used Isolation forest algorithm to extract 5% of the ...
1
vote
1answer
63 views

What are the techniques for anomaly detection of Unsupervised learning problem

I have sufficient and properly formatted data in millions without labels. I have to find out the anomalies. Heard Isolation forest, Mahalanobis distance about identifying anomalies in unsupervised ...
0
votes
1answer
39 views

Skewed two class data set

Is there any theory on the influence of skew in the data set on the performance of binary classifiers? At work, we are doing abuse detection, the negative population is regular logins, and the ...
0
votes
1answer
420 views

Validation loss is lower than the training loss

I am using autoencoder for anomaly detection in warranty data. Architecture 1: The plot shows the training vs validation loss based on Architecture 1. As we see in the plot, validation loss is ...
1
vote
1answer
33 views

Is train/test-Split in unsupervised learning of neural network necessary?

I am using autoencoder for anomaly detection in warranty data. It is unsupervised. I calculate the reconstruction error by the model and the records with high reconstruction error value is considered ...
-1
votes
1answer
38 views

How to use K-Means to detect users anomaly in Access Control

I'm currently working on access control project, Smart Lock to be more spesific. Like the other smart lock system, the system required user's authentication to open the door. I'm using RFID as ...
1
vote
2answers
73 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
2
votes
4answers
54 views

i"m confused that how to apply k means in my dataset

I have to detect anomalies from my data-set. The anomaly is about in which area and in which time of network usage(total_activity in my data) drastically improved. Help me to know how to apply k-means ...
1
vote
2answers
34 views

What is a good method for detection of rare occurencies of speech in noisy audio data?

I have some audio recordings (with relatively static but noisy background, e.g., wind in an open area) with small number of short occurrences of speech (~1% of the total audio duration). What would ...
1
vote
0answers
14 views

What do you call an anomaly/signature detection algorithm in data science

If I have an algorithm for detecting a set of data points that indicate with a high level of certainty that some event has occured OR that behaviors outside of a set model are occuring. What would ...
0
votes
1answer
34 views

is there a way to check if i got a “good price” on something?

I'm looking at some data. Actually, the Boston Housing dataset is probably a good proxy for it: https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html I'm wondering if there's a way to ...
2
votes
1answer
36 views

Given data that is labeled as outliers, how can I classify data as outliers?

I have a dataset that is a mixture of sparse binary features and quantitative features. I only have definite outliers labeled. How should I approach trying to classify unlabeled data? I considered ...
2
votes
2answers
4k views

Anomaly detection on time series

I've just started to working on an anomaly detection development in Python. My data sets regard a collection of timeseries. More in details, data are coming from some sensors/meters which record and ...
1
vote
2answers
163 views

Cross validation for anomaly detection using autoencoder

I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't ...
1
vote
2answers
54 views

How can I find anomalies in each row of data?

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility. ...
0
votes
0answers
23 views

How to define an anomaly detection problem?

I defined a supervised machine learning tasks as: Given: $m$ training examples with $n-1$ features: $E\in\mathcal{R}^{m\times (n-1)}$ and labels: $L \in \mathcal{R}^m$, a model $f$ of $d$ ...
1
vote
1answer
108 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of ...
1
vote
1answer
135 views

Using DONUT algorithm with keras

I am trying to get this repo of Xu's DONUT algorithm running, however I am getting an error I do not quite understand. The readme says I should load raw_data as follows: ...
1
vote
2answers
291 views

ML Algorithm for anomaly detection in paired time-series

I have many (around 40) separate time-series from different sensors, each measuring magnetic field intensity. I am looking to get an ML algorithm to identify a particular anomaly. This anomaly happens ...
4
votes
1answer
2k views

Using an autoencoder for anomaly detection on categorical data

Say a dataset has 0.5% of its features continuous and 99.5% categorical (binary) with ~2400 features in total. In this dataset, each observation is 1 of 2 classes - Fraud (1) or Not Fraud (0). ...
3
votes
0answers
705 views

Isolation Forest Feature Importance

As of scikit-learn version 0.19.1, there is no implementation for calculating feature importance in an Isolation Forest. I'm also having trouble finding any online resources proposing ways to get at ...
3
votes
1answer
3k views

How would I apply anomaly detection to time series data in LSTM?

I am using a LSTM RNN in Python and have successfully completed the prediction phase. My ultimate goal is anomaly detection. I'm hoping to have something like what you could see on Facebook Prophet, ...
0
votes
0answers
99 views

Machine Learning Algorithm for Dynamic Environments

Which methods are best for managing and predicting and labeling data in dynamic environment? The system data distribution changes and it is not static. The system can have different normal settings ...
0
votes
4answers
694 views

Multivariate Time Series Anomalous Entry Detection

I have a multivariate data set of the following structure. It is a time series sequence of logs with additional string attribute columns id1 and id2. If too many entries come in a sequence that have ...
0
votes
1answer
375 views

Python: Detect if data of a time series stays constant, increases or decreases

i need to analyse and later try to improve (integrate a filter) for measurement data that i compare to accurate reference data with python. First i want to calculate the mean offset and the standard ...
2
votes
2answers
654 views

Outlier detection with sklearn

I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the ...
1
vote
1answer
263 views

Isolation forest: how to deal with identical values?

I am trying to develop my own implementation of isolation forest algorithm. However I don't know how to deal with points that have the same value for a given feature. To better understand the problem, ...
3
votes
1answer
371 views

Multi Class + Negative Class Image Classification Strategies

I have seen a recurring theme in real-world problems I've worked with, where the problem looks something like "build an image classifier that recognizes classes A, B, and C but if the input is not ...
5
votes
2answers
3k views

Looking for good package for anomaly detection in time series

Is there any comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series? There is a one class SVM package in scikit-learn but it is not for ...
2
votes
1answer
77 views

Dynamic clustering

I am performing anomaly detection on different datasets and thought to first cluster the dataset and submit each of the clusters to different AD models. I am using HDBSCAN, and in my test dataset I ...
1
vote
2answers
368 views

Using local outlier factor score to detect outliers at run time

I am using LOF ( local Outlier factor) to detect outliers in my data. I get LOF score as outlier distance. this unsupervised learning doesnt help to detect outliers at run time. So I want to use my ...
1
vote
1answer
658 views

Anomaly detection with time series

I have to study the behavoiur of a machine during its works: I have available time series of pressure, temperature and other physical measures for each work it is performing, I would like to predict ...
1
vote
0answers
62 views

Detection of Anomalous Sequential User Behavior

I have a dataset containing a set of normal user sessions. Each session contains a suite of ordered user requests on N system resources {R1, ..., RN}. I want to design a continuous authentication ...
0
votes
1answer
41 views

How to represent relation between users as a feature?

I'm developing a model for unsupervised anomaly detection. I have a dataset representing communications between users (each example represents a communication): there are many features (time, duration,...
1
vote
1answer
78 views

Principal Component Analysis and abnormal data

I know that PCA is good in differentiating between anomalies and normal data and it helps to differentiate between them when it tries to transfer the data to another dimension. I mean it can somehow ...