Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

VAE latent space dimension with Weibull PDFs as input data

I am currently using a Variational Autoencoder to reconstruct PDFs of Weibull distributions with varying parameters (sampled uniformly from a given parameter span). The PDFs are generated by sampling ...
0
votes
0answers
8 views

LSTM based anomaly detection scheme too closely tracking long spans of anomalous points

I've built a time series anomaly detection process that accurately predicts the value at the next interval. However, when there are dozens of anomalous events in a row, the model starts to "catch up" ...
0
votes
0answers
11 views

Functions in scikit that detect outliers automatically?

I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?
0
votes
2answers
42 views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
0
votes
0answers
18 views

dataset shift, covariate shift or sample selection bias in small subsets?

I try to show from which training data size on which machine learning method (CNN, SVM etc.) achieves better performance. For this I would like to use subsets of different sizes from the datasets of ...
0
votes
1answer
28 views

Is it safe to use labels created from unsupervised model to train a supervised model using the same data?

I have a dataset where I have to detect anomalies. Now, I use a subset of the data(let's call that subset A) and apply the DBSCAN algorithm to detect anomalies on set A.Once the anomalies are detected,...
3
votes
0answers
23 views

Detecting abundance of a certain periodic pattern in a time series?

I am really stumped at the moment about how to solve a particular problem. I have many time series like this: This represents the number of hours a person spends on a website each day throughout the ...
1
vote
2answers
46 views

anomaly detection - conceptual help on machine learning

I am working on Anomaly detection model problem for a finance data set - set of card transactions. My team member suggested an idea that " First train the model with normal instances of data and to ...
0
votes
1answer
20 views

How to implement Classification and Anomaly detection (C++)

I am creating a system using C++(DX11) and i'm reading raw data into my program, i want to classify what the 3D data-set i'm reading in is and detect any anomalies it may have when compared to a ...
0
votes
0answers
18 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
0answers
8 views

How to solve a classification problem with multiple time series?

I am trying to build a model for credit default prediction. I've got a dataset of over 20,000 customers and the features are their payments over the last ≤24 months. The dataset looks like this: <...
0
votes
2answers
35 views

Dealing with informative missingness

How can I deal with a time series that contains missing data which means something? So the value that is missing is not wrong. It's missing on purpose and imputing those missing values would mean a ...
0
votes
1answer
30 views

Anomaly detection for time series data with only positive samples?

I'm having a time series ECG dataset. I want to do anomaly detection (anything different from normal ECG should be abnormal). The point is I'm having only positive samples with very few negative ...
0
votes
2answers
48 views

Anomaly Detection in Time Series: How to label the data

How to label time series so that we can train it on machine learning models to classify data point as anomaly or not? If I have time series, and anomaly occurs at time t, should I label that point 1 ...
0
votes
0answers
11 views

QuantileTransform before or after long-to-wide format transformation?

I perform the following steps: Perform quantile transform Transform it from wide to long format Train the anomaly detection model on the wide-format data Will there be some significant change if I, ...
0
votes
1answer
33 views

CNN for unsupervised anomaly detection

I'm wondering if the following strategy has been already used and could work Let's says you have a CNN which work well to classify image data, dog and cat. You only have cat and dog image as training ...
1
vote
2answers
33 views

Comparing two time series data to find deviations between them [closed]

This is a use case that I have and I am trying to automate this. Any pointers would be helpful. Use Case: When we deploy any new version of a web service, we keep monitoring it (while deploying to ...
1
vote
1answer
27 views

Is it meaningful to use word2vec for non-string inputs like time series analysis?

I am working on a project that detects anomalies in a time series. I wonder if I can use word2vec for anomaly detection for non-string inputs like exchange rates?
1
vote
0answers
17 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
0
votes
0answers
234 views

Anomaly detection using k-means clustering in Python

I'm working on an anomaly detection task in Python. Datasets regard a collection of time series coming from a sensor, so data are timestamps and the relative values. In order to find anomalies, I'm ...
0
votes
1answer
28 views

Anomaly detection in a database

We have a production database. The load on the database varies at different times. I want to identify anomalies; for e.g, the number of database processes responding to user queries at 9 am is 100 for ...
0
votes
0answers
19 views

Handling Multiple Classes in Categorical variables and Modeling help

The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories. I tried doing one-hot encoding on the ...
0
votes
0answers
68 views

How do we overfit a CNN AutoEncoder for anomaly detection?

I have been working on an anomaly detection problem in which I need to treat the images of "street" as an anomaly. The images of "glaciers" will be treated as not-anomaly. The autoencoder which ...
0
votes
1answer
36 views

What kind of model is this?

Can anyone help me identify what kind of Architecture is behind this Application? Is it a "simple" Classification Network? If so how are the heatmaps generated? https://www.youtube.com/watch?v=...
0
votes
0answers
38 views

How to detect anomalies (errors and exceptions) in log files?

Is this a good approach? So I'm working on a Root Cause Analysis system which should help find the cause/the root error of failed system builds (packaged in a tarball), through the analysis of log ...
0
votes
1answer
35 views

Pandas datastructure

I'm trying to analyze database performance over a period of time and detect anomalies. The database server consists of many threads that perform different actions. I run a query to determine the ...
0
votes
0answers
39 views

Using smote for data augmentation of a data set which has no dependent variable

I am trying to use the reconstruction error obtained using an auto encoder to do novelty detection. My data set is of size (4500,55)(Note: this data doesn't have any abnormalities.When an auto-encoder ...
1
vote
0answers
59 views

Can autoencoders take time series into account?

Here, I read the following: The first key to understanding is that HTM relies on data that streams over time (...) By contrast, conventional deep learning uses static data and is therefore time ...
2
votes
0answers
47 views

unsupervised outliers detection - possible solution?

I have dataset of traders' transaction data: trade id, date, stock id, sector of stock id, buy-or-sell, volume $ The goal is to identify anomalies in transactions data of traders. For example to ...
4
votes
1answer
339 views

Isolation forest sklearn contamination param

I'm working on an unsupervised anomaly detection task on time series using isolation forest algorithm. I'm developing in Python, more in detail using sklearn. I found out a lot of examples on this, ...
0
votes
0answers
15 views

A way to identify anomalous trends amongst several trends?

I'm working on a personal project, where I'd like to identify anomalous trends. Here's the scenario: Imagine a company can sell 3 types of say, candies: X, Y, and Z. For some reason, these prices can ...
0
votes
0answers
14 views

Script Response Time Anomaly Detection

Currently, I am figuring out how to tackle this problem. I have a dataset of response times for various scripts. These scripts each have multiple steps, of which each step has multiple sub-steps. ...
-1
votes
2answers
142 views

Anomaly detection k-means in Time Series

I'm trying to use k-means to detect anomalies in the Amount column. I have the following part of my dataset: ...
1
vote
2answers
23 views

How to find vertical clusters in 1-D data

I have residuals of a multivariate time series data obtained from sensors on a server.spikes in the plots of residuals indicate abnormal server state. I want to cluster the data into vertical clusters ...
0
votes
1answer
18 views

How to identify new clusters that the training data has never seen

I have to identify the different operational states of a server. I have readings related to the different sensors of the server ( like temp sensor,fan speed sensor,job load sensor etc).The data I have ...
1
vote
1answer
35 views

Get how similarity between the training data and the income data?

I'am trying to use Clustering and Classification methods as SVM using scikitlearn. I'm also studying some outliers/novelty detections I want something like a semi-supervised model. I want to predict ...
2
votes
5answers
165 views

Time Series:Outlier Detection

I have time series data which looks like the graph mentioned below. I am familiar with the method of removing outliers based on the standard deviation and median values. Drawback of these methods are ...
2
votes
5answers
165 views

Methods to detect this kind of outliers

Background I don't know much about (or to say anything) about data science or machine learning. But I'm interested in learning and thought this problem can be solved with machine learning. That's why ...
0
votes
1answer
39 views

Median absolute deviation vs standard deviation

Can Gaussian density distributions be modified using median and median absolute deviation as opposed to mean and standard deviation (since the former are more robust)?
2
votes
0answers
91 views

Out of stock / Spike in demand prediction

The goal is to predict out-of-stock situations, either quantitatively (the gap) or qualitatively (out-of-stock likely to happen in next few weeks). Background: We have existing demand planning ...
0
votes
0answers
40 views

Image anomaly detection

I am using a conv net to classify 20 different patterns on an image. My train/test set are images where I know the class. With this, I a good enough model for my preference. It's an unbalance dataset, ...
0
votes
1answer
26 views

Clustering unbalanced dataset

The data I am working on has some really large price values and some really small values. What I did was first perform feature bagging on the data and got them labelled to (0,1) and then did ...
0
votes
0answers
33 views

Methodologies and Trends for Sequence Anomaly Detection

I recently started to approach the issue of detecting deviating behavior from rule-based sequences. Basically the task is to spot any difference from "normal running" of processes that are defined as ...
1
vote
2answers
71 views

Outlier detection for Disk Space Usage

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset): ...
0
votes
3answers
86 views

What kind of learning in this training situation when anomaly detection? Supervised learning,semi-supervised learning or unsupervised learning?

I am doing anomaly detection recently, one of the methods is using AEs model to learn the pattern of normal samples. Determine it as an abnormal sample if it doesn’t match the pattern of normal ...
0
votes
0answers
56 views

Anomaly detection in time series data from multiple sensors

I've build a classification model based on 15 features coming in real time from 15 different sensors. The window time is 60 seconds (which means that the classification model needs 60 records from ...
0
votes
0answers
27 views

Which unsupervised learning algorithm can be used for peaks detection?

So, I have a dataset which has around 1388 unique products and I have to do unsupervised learning on them in order to find anomalies (high/low peaks). The data below just represents one product. The <...
0
votes
2answers
63 views

A Proper Outlier Detection For the Attached Figure

I am wondering how I could detect the red points as outliers using an algorithm (best method for this scenario) not through visualization since it is clear that they are outliers in the figure. ...
1
vote
1answer
108 views

Are Denoising Autoencoders for anomaly detection on structured data?

Can denoising autoencoders be used for anomaly detection on structured data? I know I can use denoising autoencoders for anomaly detection on images, but I don't know if they can do it for structured ...
1
vote
1answer
33 views

How to construct confidence bound for Time Series Prediction?

I have some time series data and am using some Deep Learning techniques to get its prediction. Now, I would like to construct confidence bounds for it. I calculated the residuals, their mean and ...