Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [anomaly-detection]

The tag has no usage guidance.

0
votes
0answers
15 views

Outliner detection with LSTM autoencoder

I am learning about autoencoders for outlier detection. I have searched enough and internet suggest to use LSTM autoencoders for outlier detection from multivariant time series data. I have watched ...
1
vote
1answer
14 views

Anomaly detection in nominal big data

I have to apply an anomaly detection algorithm on big data, the values of each column on my dataframe are nominal and vary over 10000 times, the algorithms I've found only accept numeric values, is ...
-1
votes
1answer
41 views

Finding outliers from multiple files [on hold]

I am dealing with a very strange problem. I have a lot of files. I need to show which files are similar and which one has exception/outliers using its data. I can show with unsupervised learning ...
1
vote
1answer
16 views

Do anomalous input features to autoencoder result in high errors on the corresponding output features?

An autoencoder is trained by replicating each training instance to both input and output. However, when predicting for anomaly detection, will the output error be local to the same output feature(s) ...
0
votes
0answers
11 views

Practical examples/tutorials of using One-Class Support Vector Machines

I am a newbie in machine learning, and hope to solve an anomaly detection task using One-Class Support Vector Machines (OCSVM). ...
0
votes
0answers
29 views

Which between a DL autoencoder, Twitter STL+ESD and Numenta HBM is more state-of-the-art for anomaly detection over timeseries?

I aim to develop a generic algorithm to detect abnormalities in time series (ideally accepting multivariate series, and the presence of some noise). The aspects which I am considering for comparing ...
0
votes
1answer
27 views

How can I detect anomalies/outliers in my online streaming data on a real-time basis?

Say, I've a huge set of data(infinite in size) consisting of alternating sine wave and step pulses one after the other. What I want from my model is to parse the data sequence wise or point wise and ...
0
votes
1answer
27 views

Sequences of time series data with only 1 output classification

So I'm facing a problem where I have a sequence (30h of data with 10sec intervals) and which is labeled to a class (3 different classes) for the whole sequence. I'm used to work with time series who ...
0
votes
1answer
21 views

Explaination of the anomalies detected

I have a dataset with a mixture of categorical and numeric variables. I have converted categorical variables into one hot encoding. I have used Isolation forest algorithm to extract 5% of the ...
1
vote
1answer
38 views

What are the techniques for anomaly detection of Unsupervised learning problem

I have sufficient and properly formatted data in millions without labels. I have to find out the anomalies. Heard Isolation forest, Mahalanobis distance about identifying anomalies in unsupervised ...
0
votes
1answer
31 views

Skewed two class data set

Is there any theory on the influence of skew in the data set on the performance of binary classifiers? At work, we are doing abuse detection, the negative population is regular logins, and the ...
0
votes
0answers
6 views

Leave one out for outlier remove in panel data

I'm working on a class project which involves calculating total sales for a given clothing brand from a sample retailer panel. I have monthly sales totals, and I want to estimate quarterly sales for ...
0
votes
0answers
36 views

Grid search for autoencoder model

I'm using autoencoder for anomaly detection. Model's output is basically the reconstructed input based on the original input. The anomalies are then calculated using the reconstrction error. I would ...
0
votes
1answer
77 views

Validation loss is lower than the training loss

I am using autoencoder for anomaly detection in warranty data. Architecture 1: The plot shows the training vs validation loss based on Architecture 1. As we see in the plot, validation loss is ...
1
vote
1answer
31 views

Is train/test-Split in unsupervised learning of neural network necessary?

I am using autoencoder for anomaly detection in warranty data. It is unsupervised. I calculate the reconstruction error by the model and the records with high reconstruction error value is considered ...
-1
votes
1answer
33 views

How to use K-Means to detect users anomaly in Access Control

I'm currently working on access control project, Smart Lock to be more spesific. Like the other smart lock system, the system required user's authentication to open the door. I'm using RFID as ...
1
vote
2answers
63 views

huge doubt on anomaly detection

from the naked eye itself, we can tell in the region 5161 the network usage is high so that is the anomaly in my case, then why do we want to apply k-means and other machine learning algorithms to ...
2
votes
4answers
49 views

i"m confused that how to apply k means in my dataset

I have to detect anomalies from my data-set. The anomaly is about in which area and in which time of network usage(total_activity in my data) drastically improved. Help me to know how to apply k-means ...
1
vote
2answers
32 views

What is a good method for detection of rare occurencies of speech in noisy audio data?

I have some audio recordings (with relatively static but noisy background, e.g., wind in an open area) with small number of short occurrences of speech (~1% of the total audio duration). What would ...
1
vote
0answers
12 views

What do you call an anomaly/signature detection algorithm in data science

If I have an algorithm for detecting a set of data points that indicate with a high level of certainty that some event has occured OR that behaviors outside of a set model are occuring. What would ...
0
votes
1answer
28 views

is there a way to check if i got a “good price” on something?

I'm looking at some data. Actually, the Boston Housing dataset is probably a good proxy for it: https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html I'm wondering if there's a way to ...
2
votes
1answer
36 views

Given data that is labeled as outliers, how can I classify data as outliers?

I have a dataset that is a mixture of sparse binary features and quantitative features. I only have definite outliers labeled. How should I approach trying to classify unlabeled data? I considered ...
1
vote
2answers
2k views

Anomaly detection on time series

I've just started to working on an anomaly detection development in Python. My data sets regard a collection of timeseries. More in details, data are coming from some sensors/meters which record and ...
1
vote
2answers
120 views

Cross validation for anomaly detection using autoencoder

I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't ...
1
vote
2answers
48 views

How can I find anomalies in each row of data?

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility. ...
0
votes
0answers
19 views

How to define an anomaly detection problem?

I defined a supervised machine learning tasks as: Given: $m$ training examples with $n-1$ features: $E\in\mathcal{R}^{m\times (n-1)}$ and labels: $L \in \mathcal{R}^m$, a model $f$ of $d$ ...
0
votes
1answer
60 views

Anomaly detection using clustering of highly correlated Categorical data

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of ...
1
vote
1answer
104 views

Using DONUT algorithm with keras

I am trying to get this repo of Xu's DONUT algorithm running, however I am getting an error I do not quite understand. The readme says I should load raw_data as follows: ...
1
vote
2answers
219 views

ML Algorithm for anomaly detection in paired time-series

I have many (around 40) separate time-series from different sensors, each measuring magnetic field intensity. I am looking to get an ML algorithm to identify a particular anomaly. This anomaly happens ...
4
votes
1answer
1k views

Using an autoencoder for anomaly detection on categorical data

Say a dataset has 0.5% of its features continuous and 99.5% categorical (binary) with ~2400 features in total. In this dataset, each observation is 1 of 2 classes - Fraud (1) or Not Fraud (0). ...
1
vote
0answers
490 views

Isolation Forest Feature Importance

As of scikit-learn version 0.19.1, there is no implementation for calculating feature importance in an Isolation Forest. I'm also having trouble finding any online resources proposing ways to get at ...
1
vote
1answer
2k views

How would I apply anomaly detection to time series data in LSTM?

I am using a LSTM RNN in Python and have successfully completed the prediction phase. My ultimate goal is anomaly detection. I'm hoping to have something like what you could see on Facebook Prophet, ...
0
votes
0answers
84 views

Machine Learning Algorithm for Dynamic Environments

Which methods are best for managing and predicting and labeling data in dynamic environment? The system data distribution changes and it is not static. The system can have different normal settings ...
0
votes
3answers
414 views

Multivariate Time Series Anomalous Entry Detection

I have a multivariate data set of the following structure. It is a time series sequence of logs with additional string attribute columns id1 and id2. If too many entries come in a sequence that have ...
0
votes
1answer
264 views

Python: Detect if data of a time series stays constant, increases or decreases

i need to analyse and later try to improve (integrate a filter) for measurement data that i compare to accurate reference data with python. First i want to calculate the mean offset and the standard ...
1
vote
2answers
510 views

Outlier detection with sklearn

I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the ...
1
vote
1answer
229 views

Isolation forest: how to deal with identical values?

I am trying to develop my own implementation of isolation forest algorithm. However I don't know how to deal with points that have the same value for a given feature. To better understand the problem, ...
3
votes
1answer
300 views

Multi Class + Negative Class Image Classification Strategies

I have seen a recurring theme in real-world problems I've worked with, where the problem looks something like "build an image classifier that recognizes classes A, B, and C but if the input is not ...
4
votes
2answers
2k views

Looking for good package for anomaly detection in time series

Is there any comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series? There is a one class SVM package in scikit-learn but it is not for ...
2
votes
1answer
72 views

Dynamic clustering

I am performing anomaly detection on different datasets and thought to first cluster the dataset and submit each of the clusters to different AD models. I am using HDBSCAN, and in my test dataset I ...
1
vote
2answers
286 views

Using local outlier factor score to detect outliers at run time

I am using LOF ( local Outlier factor) to detect outliers in my data. I get LOF score as outlier distance. this unsupervised learning doesnt help to detect outliers at run time. So I want to use my ...
1
vote
1answer
501 views

Anomaly detection with time series

I have to study the behavoiur of a machine during its works: I have available time series of pressure, temperature and other physical measures for each work it is performing, I would like to predict ...
1
vote
0answers
58 views

Detection of Anomalous Sequential User Behavior

I have a dataset containing a set of normal user sessions. Each session contains a suite of ordered user requests on N system resources {R1, ..., RN}. I want to design a continuous authentication ...
0
votes
1answer
39 views

How to represent relation between users as a feature?

I'm developing a model for unsupervised anomaly detection. I have a dataset representing communications between users (each example represents a communication): there are many features (time, duration,...
1
vote
1answer
74 views

Principal Component Analysis and abnormal data

I know that PCA is good in differentiating between anomalies and normal data and it helps to differentiate between them when it tries to transfer the data to another dimension. I mean it can somehow ...
4
votes
1answer
1k views

Multivariate outlier detection with isolation forest..How to detect most effective features?

I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in http://scikit-learn.org/stable/auto_examples/ensemble/...
2
votes
2answers
92 views

General methods outlier detection

What are general methods for outlier detection that do not assume any underlying distribution in the data? I have a dataset with the prizes of the rents in London, as well as their location, number of ...
3
votes
0answers
285 views

Anomaly detection in cooling process data without exact labels

I have a data set where I look at the cooling of a process. The starting temperature may vary between 580 and 180 degrees. I know that at some point the cooling system failed (see examples in the plot)...
1
vote
1answer
203 views

Anomaly Detection from available sensor data set?

I am working on live sensor data set and looking to find abnormal(leads to machine fault condition) pattern form available data set. I am learning and new to this world of Data science, but comfort ...
0
votes
1answer
136 views

outliers detection with non normal distribution

What are some techniques I can use for anomaly detection given a non-Normal distribution? I have less than twenty available observations.