Questions tagged [anomaly-detection]

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behaviour. This is also known as outlier detection.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Explanation of Excess Mass(EM)

I was researching on evaluation metrics to understand the performance of unsupervised anomaly detection algorithms and I came across this paper The author suggests that EM and MV based numerical ...
7
votes
1answer
491 views

Learning with Positive labels only

I have ~7 million rows of customer data (~500 sparse attributes) A million out of them have opted in to a new service. How do I use this signal to predict which of the remaining customers are likely ...
1
vote
1answer
18 views

Is window based sequencing a good idea to obtain more training data for LSTMs?

I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier. However,...
2
votes
2answers
22 views

Log analysis dataset with labeled cybersecurity issues

I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but ...
0
votes
0answers
14 views

Can I treat concept drift detection as a binary clarification problem?

Assume I have the ground truth labels for both non-drited and drifted samples, can I treat concept drift detection as a binary classication problem (one class non-drifted one class drifted)? If not ...
0
votes
2answers
24 views

Looking for spatial clusters and anomalies. Is DBSCAN the right tool?

I have a regular 2D grid of data points (X, Y) with each point having a value. I'd like to identify clusters and then anomalies that don't belong to those clusters. I'm trying to understand the best ...
2
votes
1answer
20 views

Word representation that gives more weight to terms frequent in corpus?

The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I ...
0
votes
1answer
19 views

Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages. My dataset is therefore a 3D array of size(m,t,N)...
0
votes
0answers
15 views

Identifying anomalies in spatial (latitude, longitude, time) data? [duplicate]

I'll start off by prefacing this with the fact that I'm not even sure if I'm asking the right question and what I'm really looking for is some guidance to dig in the right direction. I have a set of ...
0
votes
1answer
24 views

Incorrect multi-variate anomaly detection - Isolation Forest Python

My data looks like below. it has 333 rows and 2 columns. Clearly the first row is anomaly. ndf: ...
0
votes
0answers
7 views

Is there any inductive Graph Variational Auto Encoder?

I have been reading about how we can model a Variational AutoEncoder (VAE) into a Graph Variational AutoEncoder (GVAE) where the decoder reconstructs the adjacency matrix. I presume that this approach ...
0
votes
1answer
27 views

Cross-Validation in Anomaly Detection with Labelled Data

I am working on a project where I train anomaly detection algorithms Isolation Forest and Auto-Encoder. My data is labelled so I have the ground truth but the nature of the problem requires ...
1
vote
0answers
17 views

Multivariate Gaussian distribution - Covariance vs linear dependence

From prof. Andrew Ng's Multivariate Gaussian distribution lecture, covariance measures linear dependency between features, in which case we might use Multivariate Gaussian distribution with covariance ...
0
votes
0answers
44 views

Anomaly Detection over multivariate data containing Nominal and numerical predictors

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors. Dataset has following pattern: If we consider the below sample records, category_id, ...
-1
votes
1answer
27 views

Does Anomaly Detection Algorithm works when the features are not correlated?

I am working on an Anomaly Detection Problem and the algorithm I used is an Autoencoder Multivariate Gaussian. The problem with my data is that it is unlabeled and not correlated. For example, let's ...
2
votes
1answer
82 views

Interpretation of scikit-learn one class svm scores

How can I interpret the scores generated by the function score_samples(X) from a scikit-learn OneClassSVM model? Is there a way ...
0
votes
1answer
39 views

Semi-supervised anomaly detection

I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods. Now, the thing is, there might ...
1
vote
2answers
36 views

Identify the parameter causing the anomaly in a multivariate dataset

I have a payment transaction dataset with a large number of predictor variables. I am trying to build a model for anomaly detection and I have evaluated various algorithms/approaches for the same like ...
2
votes
0answers
29 views

What's the best way to validate a rare event detection model during training?

When training a deep model for rare event detection (e.g. sound of an alarm in a home device audio stream), is it best to use a balanced validation set (50% alarm, 50% normal) to determine early ...
1
vote
0answers
81 views

Hyperparameter tuning one-class svm

I have a problem where I am trying to apply a one-class svm to detect outliers. I am training on a dataset of true cases using a one-class radial svm and then predicting for both false and true cases. ...
2
votes
0answers
29 views

anomaly detection in vehicle sensor data

I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors. It seems like there is no proper book or article that goes deeper into ...
0
votes
0answers
23 views

Which graph to choose to plot anomaly detection

I'm creating tool to detect anomalies in syslog messages. I'm parsing syslogs into Bag of Words of 200 features. This BoW is forwarded into machine learning model based on selection. I got 12 models - ...
0
votes
1answer
46 views

Can we use one hot encoding instead of for loops?

I have an anomaly detection model, that I run per store with a bunch of features. I intend to run this code, everyday, per store. Now, lets say I have 8000 stores, I would imagine, I should write a ...
2
votes
0answers
17 views

Types of artificial anomalies

I am working on some algorithms for anomaly detection The dataset is clean our anomalies so I want to add some artificial anomalies. I have added some anomalies. I get the maximum value of the ...
0
votes
0answers
58 views

Isolation forest challenge with contamination factor

I do isolation forest with time series data for anomaly detection. Its unsupervised model which detects anomaly with past 2 weeks data on todays data the window moves forward everyday Due to auto ...
0
votes
2answers
114 views

What does the classification report interpret? Class 1 indicates abnormal data

How to interpret the report and How is precision, recall values are calculated for individual class labels. What is the significance of macro avg ? Does this report signify a good predictions by the ...
1
vote
0answers
28 views

Classifier for DBSCAN [closed]

I have written a code that uses DBSCAN and tries to find the most appropriate eps for my dataset, trying to include most of the data inside a cluster. The problem is that DBSCAN is not a classifier ...
3
votes
1answer
47 views

Outlier/Anomaly Detection History

I have been reading about different methods of anomaly detection, their structure and the way they work. Recently I have been trying to find some scholar articles, writings or books where I can learn ...
0
votes
0answers
8 views

Anomalies timeseries: discard feature vectors or samples?

I'm implementing a classifier which classifies based on a time-series. This time-series is made out of a 50Hz sensor and measures three items (x, y, z). Yet sometimes there's an unusually high value, ...
0
votes
0answers
19 views

Is there any know algorithm or technique to detect IOT clock anomaly

I have some IOT machine in place and a user have to present a NFC card on it when using one of this machine. When the machine is detecting the card it send an event to a backoffice system with the ...
0
votes
0answers
11 views

Reporting false alarm rate and detection rate in shilling attack detection without given any labels

I have a dataset of ratings given to movies by users. I've applied the method mentioned in this paper to detect fake votes(I've used $H(X)$ and $M(X)$ measures). But the dataset I'm using doesn't have ...
1
vote
1answer
32 views

SPC vs Autoencoders in anomaly detection

Considering the usage of Autoencoders in anomaly detection of time-series data, why SPCs (control charts) have lost their charm? Are there any advantages with Autoencoders and disadvantages with SPCs?
0
votes
0answers
22 views

Custom Decision Function for Custom Outlier Detection Algorithm

I have built a custom algorithm for semi-supervised anomaly detection and here is my output example as following with probability threshold set to 0.05 and 1 = outlier, 0 = inlier: ...
0
votes
0answers
13 views

How do I evaluate a K-Means unsupervised anomaly detection approach?

how do I evaluate K-means clustering anomaly detection method as there is no labelled data of anomaly class. To find the cluster (K), I have used the silhouette score from Scikit learn library. Scikit ...
2
votes
0answers
53 views

How Yelp System Detects Paid Reviews

I am wondering how the yelp spam detection system detects paid reviews? By paid, I mean the following scenarios: I as a business owner pay people to write positive comments and give me a good rate I ...
0
votes
0answers
22 views

is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most ...
0
votes
0answers
19 views

Clustering a dataset and creating a model per each cluster

I was wondering if it makes sense to cluster a dataset to find closely related data points and train a binary classification model for each of this clusters as they would be minidatasets. I'll ...
0
votes
2answers
102 views

Percentile as a threshold for Anomaly Detection?

I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is ...
-1
votes
2answers
92 views

Plotting ROC & AUC for SVM algorithm

Towards , the end of my program, I have the following code. model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00001) model.fit(train_data) Output ...
2
votes
2answers
67 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
2
votes
3answers
57 views

K-Means anomaly detection not clustering anomalies

K-means anomaly detection scatter plot The following code, takes a single column from a dataset and then adds 50 anomalies to the dataset that is quite bigger than the maximum values of the dataset. ...
0
votes
1answer
15 views

How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python)

I have some data which contains anomalies as well. I want to model data using Gaussian distribution assuming variables are dependent in Python. How can I model this? Should I use the PDF formula as ...
0
votes
0answers
12 views

How to set anomaly threshold depending of predictive model accuracy

Say I have a variable with a standard deviation STD I have a predictive model to predict variable. The model accuracy is 80% An anomaly is raised if difference (predicted_value - actual value) > ...
1
vote
0answers
27 views

Anomaly Detection Methods for Clean Training Data [closed]

The goal is to find anomalies in my dataset, univariant anomaly detection which basically means looking for anomalies only in one column. Up until this point I have tried DBSCAN; Isolation Forest and ...
2
votes
1answer
43 views

Adding anomalies to the Dataset

Recently I have been trying different Scikit-Learn anomaly detection clustering methods, like DBSCAN Isolation Forest. Based on how many training data I use, how I tweak on the algorithms ...
0
votes
0answers
55 views

Online courses for Anomaly Detection

As the title say I have been looking for some online courses that would teach me about anomaly detection using Unsupervised Machine Learning. I want to focus only on Scikit-Learn and not go deeper ...
2
votes
1answer
84 views

Outlier Detection using K-Means using one column

I have done and read a csv file and then plotted the values of a single column using K-means ...
0
votes
0answers
23 views

box plot and the anomaly detection?

i'm doing a classification problem with 50000 rows × 5000 columns of dataset. calssification label is 300 labels 1. This is the few example of box plot of some feature. what can I inference from ...
0
votes
0answers
20 views

PyOD algorithms that support datasets with empty cells

The idea is to find outliers in large datasets. What am I going to use to detect these outliers? I am going to use a library called PyOD for the detection of anomalies which was developed by Yue ...
1
vote
1answer
42 views

Effects that Empty cells have in Unsupervised Machine Learning

I have a dataframe in a csv file that I would like to perform different unsupervised machine learning algorithms on. The file itself has some cells that are not ...

1
2 3 4 5